Text Retrieval Conference Series 

Supports Information Retrieval 

Research Community

Update on NIST Cybersecurity 

Framework

ITL and IEEE Collaborate on 

Election Results Reporting 

Standard

ITL Staff Tutorials on Proficiency 

Testing

Wounded Warrior Presentation at 

NIST

Staff Accomplishments

Selected New Publications

Upcoming Technical 

Conferences

TEXT RETRIEVAL CONFERENCE SERIES SUPPORTS 

INFORMATION RETRIVAL RESEARCH COMMUNITY 

Started in 1992 and cosponsored by the Department of Defense, ITL’s Text REtrieval 

Conference (TREC) series of evaluation workshops is designed to support the information 

retrieval community by providing the infrastructure necessary for large-scale evaluation of text 

retrieval methodologies. Over the years, the workshops have evolved based on emerging 

information retrieval technologies. The evaluation effort has grown in both number of 

participating systems and the number of countries represented.

Each TREC is organized around a set of focus areas called tracks. TREC participants use 

their own search engines and a common data set to perform a track's task. They submit their 

search results to NIST, which uses the combined result sets from all participants to build 

evaluation resources that are then used to score each participant's submission. Both the 

evaluation resources and the participants' submissions are made publicly available through 

the TREC website to support the larger retrieval research community. 

TREC 2014, the 23rd conference in the series, was held on November 19-21, 2014. The 

conference included eight tracks that drew 75 participating teams representing 20 countries. 

The tracks investigated topics ranging from risk minimization in web search to search over 

microblogs to efficiently monitoring the information associated with an event such as a natural 

disaster in real time.

TREC 2014 added an exciting new track called the Clinical Decision Support (CDS) track. The 

goal was to develop systems that can support healthcare providers by finding clinically 

relevant information within the vast biomedical literature. This initial year of the track used 

case narratives developed by physicians at the National Library of Medicine as patient 

surrogates. A case report is a well-formed narrative summarizing portions of a patient's 

medical record that typically describes a challenging medical case. The literature base was 

the open-access portion of PubMed Central. The systems' task was to retrieve articles that 

contained pertinent information for one of three clinically relevant questions for the target 

patient: What is the diagnosis? What test should be performed? What treatment should be 

undertaken? Twenty-six teams participated in the CDS track. The retrieval results suggest that 

the task is challenging, but feasible, for existing search systems. See the TREC website for 

more information.
2

Update on NIST Cybersecurity 

Framework

Under Executive Order (EO) 13636, Improving Critical 

Infrastructure Cybersecurity, NIST was tasked to work with 

stakeholders to develop a voluntary framework for 

managing cybersecurity risks to critical infrastructure. NIST 

released version 1.0 of the Framework on February 12, 

2014. The Framework allows organizations—regardless of 

sector, size, or degree of cybersecurity risk or cybersecurity 

sophistication—to apply the principles and best practices of 

risk management to improve the security and resilience of 

critical infrastructure.

ITL recently held its 6th Cybersecurity Framework 

workshop to discuss organizations’ initial experiences with 

the Framework, with a focus on resources to help 

organizations use the Framework to improve their 

cybersecurity risk management programs. The 300+ 

attendees included representatives from a variety of 

sectors, including banking, chemical, telecommunications, 

information technology, and energy. The workshop 

featured several topic-specific sessions on authentication, 

automated indicator sharing, the cybersecurity workforce, 

supply chain, standards, and the Framework’s privacy 

methodology. Participants discussed challenges and 

possible paths forward in each of these areas. See the 

framework website for additional information. 

ITL and IEEE Collaborate on Election 

Results Reporting Standard

On the recent November 4th election night, the State of 

Ohio made their election results available to the public 

using an XML standard format designed largely by ITL 

working with the IEEE Voting System Standards 

Committee (VSSC). ITL and the VSSC developed the 

standard to reduce complexity in election results collection 

and publishing, especially on election night when deadlines 

are tight and many opportunities for error exist. The 

standard specifies two data interchange formats—XML and 

JSON—for reporting data exported from equipment used to 

manage elections and tabulate results. It allows for 

reporting on election information known in advance, 

election night results, and updates and certified results 

from postelection canvassing. The standard is edited by 

ITL, and the final version is expected in early 2015. More 

information about the standard can be found at this 

website. 

ITL Staff Tutorials on Proficiency 

Testing

ITL statisticians recently presented two tutorials on 

proficiency testing at the Combined Regional Measurement 

Assurance Program (CRMAP) meeting, held in St. Louis, 

Missouri. The CRMAP meeting is organized and led by the 

NIST Office of Weights and Measures and brings together 

metrologists from the state weights and measures 

laboratories, commercial calibration laboratories, and 

metrology equipment manufacturers. The first tutorial 

covered the ISO 13528 standard for proficiency testing, 

which outlines best practices and recommended statistical 

methods for proficiency testing. The second tutorial 

covered frequently asked questions from the analysis of 

proficiency test data. A new electronic audience response 

system proved popular, gave instant feedback on the 

audience’s understanding of the material, and spurred 

further questions and discussion of the concepts being 

presented.

Wounded Warrior Presentation at NIST

The ITL Diversity Committee and 

the NIST Civil Rights and 

Diversity Office recently 

cosponsored a presentation by 

retired Staff Sergeant Erick 

Millette from the Wounded 

Warrior Project. Wounded in the 

Iraq War and subsequently 

discharged from the Army, 

Millette gave a moving and 

thought-provoking account of his 

experiences and the physical, 

mental, and emotional challenges 

he has faced since his discharge. 

The presentation raised 

awareness at NIST of the ongoing problems of returning 

veterans and the help that is available through 

organizations such as the Wounded Warrior Project.

Staff Accomplishments

Simone Gittelson, Statistical Engineering Division, received 

the Prix de la Banque Cantonale Vaudoise for her 2013 

Ph.D. thesis entitled “Evolving from Inferences to Decisions 

in the Interpretation of Scientific Evidence.” She received 

the prize at the graduation ceremony of the School of 

Criminal Justice, University of Lausanne, in Lausanne, 

Switzerland, on November 7, 2014.
3

Selected New Publications

US Government Cloud Computing 

Technology Roadmap Volume I: HighPriority Requirements to Further USG 

Agency Cloud Computing Adoption and 

Volume II: Useful Information for Cloud Adopters

By Lee Badger, David Bernstein, Robert Bohn, Frederic de 

Vaulx, Mike Hogan, Michaela Iorga, Jian Mao, John 

Messina, Kevin Mills, Eric Simmon, Annie Sokol, Jin Tong, 

Fred Whiteside, and Dawn Leaf

NIST Special Publication 500-293, Vols. I and II

October 2014

Consistent with its mission, NIST has a technology 

leadership role in support of United States Government 

(USG) secure and effective adoption of the Cloud 

Computing model to reduce costs and improve services. In 

the technology vision of Federal Cloud Computing Strategy 

success, USG agencies will be able to easily locate 

desired IT services in a mature and competitive 

marketplace, rapidly procure access to these services, and 

use them to deliver innovative mission solutions. 

Assessing Security and Privacy Controls in Federal 

Information Systems and Organizations: Building Effective 

Assessment Plans

Joint Task Force Transformation Initiative

NIST Special Publication 800-53A Revision 4

December 2014 

This publication provides a set of procedures for 

conducting assessments of security controls and privacy 

controls employed within federal information systems and 

organizations. The assessment procedures, executed at 

various phases of the system development life cycle, are 

consistent with the security and privacy controls in NIST 

Special Publication 800-53, Revision 4. The procedures 

are customizable and can be easily tailored to provide 

organizations with the needed flexibility to conduct security 

control assessments and privacy control assessments. 

Media Sanitization

By Richard Kissel, Andrew Regenscheid, Matthew Scholl, 

and Kevin Stine

NIST Special Publication 800-88 Revision 1

December 2014

Media sanitization refers to a process that renders access 

to target data on the media infeasible for a given level of 

effort. This guide will assist organizations and system 

owners in making practical sanitization decisions based on 

the categorization of confidentiality of their information.

Guidelines for Derived Personal Identity Verification (PIV) 

Credentials

By Hildegard Ferraiolo, David Cooper, Salvatore 

Francomacaro, Andrew Regenscheid, Jason Mohler, 

Sabari Gupta, and William Burr

NIST Special Publication 800-157

December 2014

This recommendation provides technical guidelines for the 

implementation of standards-based, secure, reliable, 

interoperable PKI-based identity credentials that are 

issued by federal departments and agencies to individuals 

who possess and prove control over a valid PIV Card. 

Screening for factors affecting application performance in 

profiling measurements

By David W. Flater

NIST Technical Note 1855

October 2014 

This report provides an example application of screening 

techniques in experimental computer science, including 

validation and selection of metrics and measures, the 

screening experiment itself, and supporting statistical 

methods. Together with related references, it is intended to 

encourage other computer scientists to make more use of 

established statistical methods in performance evaluations. 

Fingerprint Vendor Technology Evaluation

By Craig Watson, Gregory Fiumara, Elham Tabassi, Su 

Lan Cheng, Patricia Flanagan, and Wayne Salamon

NISTIR 8034

December 2014 

FpVTE was conducted primarily to assess the current 

capabilities of fingerprint matching algorithms using 

operational datasets containing several million subjects. 

There were three classes of participation that examined 

various finger combinations from single finger all the way 

up to ten fingers. Enrollment sets varied in size from 

10,000 subjects up to 5 million subjects. All data used was 

sequestered operational data that was not shared with any 

of the participants. The evaluation provided feedback to 

the participants after the first two of three submissions, 

allowing participants to evaluate their performance, make 

adjustments to their algorithms, and resubmit for further 

testing. The evaluation was conducted at NIST using NIST

-owned hardware. Participants submitted software libraries 

compliant to the testing Application Programming Interface 

(API), which were linked to a NIST-developed test driver 

and run by NIST employees. All participant libraries went 

through validation testing to ensure that results at NIST 

matched results participants were getting on their 

hardware. This is the first large scale one-to-many 

fingerprint evaluation since FpVTE 2003. 
The Information Technology Laboratory (ITL) is a major research component of the National Institute of 

Standards and Technology (NIST). As a world-class measurement and testing laboratory encompassing a wide 

range of areas of computer science, mathematics, statistics, and systems engineering, our research program 

supports NIST’s mission to promote U.S. innovation and industrial competitiveness by advancing measurement 

science, standards, and technology in ways that enhance economic security and improve our quality of life. ITL 

cybersecurity experts collaborate to develop cybersecurity standards, guidelines, and associated methods and 

techniques for federal agencies and industry. Our mathematicians and statisticians collaborate with 

measurement scientists across NIST to help ensure that NIST maintains and delivers the world’s leading 

measurement capability. ITL computer scientists and other research staff provide technical expertise and 

development that underpins national priorities such as cloud computing, the Smart Grid, homeland security, 

information technology for improved healthcare, and electronic voting. We invite you to learn more about 

how ITL is enabling the future of the nation’s measurement and standards infrastructure for information 

technology by visiting our website at http://www.itl.nist.gov.

ITL Editor: Elizabeth B. Lennon

National Institute of Standards and Technology

100 Bureau Drive, Stop 8900

Gaithersburg, MD 20899-8900

Phone: (301) 975-2832

Fax: (301) 975-2378

Email: elizabeth.lennon@nist.gov

TO SUBSCRIBE TO THE

ELECTRONIC EDITION OF THE

ITL NEWSLETTER, GO TO

ITL HOMEPAGE

Upcoming Technical 

Conferences

Cybersecurity for Direct Digital Manufacturing 

Symposium

Date: February 3, 2015

Place: NIST, Gaithersburg, Maryland

Sponsor: NIST

Cost: None

This symposium will explore cybersecurity needed for direct 

digital manufacturing. Speakers from industry, academia, 

and government will discuss the state of the industry, 

cybersecurity risks and solutions, and implications for 

Information and Communications Technology (ICT) supply 

chain risk management.

NIST contact: Celia Paulsen

Hands-on Workshop on Assessing and Reporting 

Measurement Uncertainty

Dates: March 18-20, 2015

Place: Anaheim, California

Sponsor: NIST

Cost: TBD

This short course covers many aspects of the propagation 

of uncertainty using the methods outlined in the JCGM 

Guide to the Expression of Uncertainty in Measurement. 

Exercises and hands-on applications will use functions for 

uncertainty analysis from the free software package, 

metRology, written for the open-source R statistical 

computing environment. 

NIST contact: Will Guthrie

FISSEA Annual Conference

Dates: March 24-25, 2015

Place: NIST, Gaithersburg, Maryland

Sponsors: NIST and FISSEA

Cost: $195 (includes coffee breaks/lunch)

$101 (no coffee breaks/lunch)

The theme of this year’s Federal Information Systems 

Security Educators’ Association (FISSEA) conference is 

Changes, Challenges, and Collaborations: Effective 

Cybersecurity Training. Attendees will gain a better 

understanding of current cybersecurity projects, emerging 

trends and initiatives, and awareness and training ideas. 

NIST contact: Peggy Himes

Workshop on Cybersecurity in a Post-Quantum World

Dates: April 2-3, 2015

Place: NIST, Gaithersburg, Maryland

Sponsor: NIST

Cost: $95 (includes coffee breaks/refreshments) $60 (no 

coffee breaks/refreshments)

The advent of practical quantum computing will break all 

commonly used public key cryptographic algorithms. In 

response, NIST is researching cryptographic algorithms for 

public key-based key agreement and digital signatures that 

are not susceptible to cryptanalysis by quantum algorithms. 

NIST is holding this workshop to discuss issues related to 

post-quantum cryptography and its potential future 

standardization.

NIST contact: Dustin Moody

Disclaimer: Any mention of commercial products or reference to commercial organizations is for 

information only; it does not imply recommendation or endorsement by the National Institute of 

Standards and Technology nor does it imply that the products mentioned are necessarily the best 

available for the purpose.

Credit: NIST

4