National Software Reference Libr

(NSRL) Project

Douglas White

Information Technology Laboratory

March 2003
Introduction

The National Software Reference Library is:• A physical collection of over 3,800 software 

packages on secured shelves

• A database of file “fingerprints” and additio

information to uniquely identify each file on shelves

• A Reference Data Set (RDS) extracted from 

database onto CD, used by law enforcementinvestigators and researchers
Law Enforcement Origins

• Mission: assist Federal, state and local agencies• NIST is a neutral organization in relation to vendors• NIST provides an open rigorous process assuring data • NIST NSRL will become an international software re

NIST

(OLES, ITL) (NIJ, FBI)

DoJ DoD

(DCFL)

SOFTWAR

VENDORS AGENCIES

NATIONAL

STATE/LOCAL

TREASURY

(USCS)
Addressing Industry Need

• No unbiased organizations were involved in 

implementing investigative tools

• Law enforcement had no control over qualit

data provided by available tools – data was market-driven

• Traceability - No repositories of original sof

available for reproducing data

• Each tool provided a limited set of capabilit
Computer Forensics in ITL

• Located in Software Diagnostics and 

Conformance Testing (SDCT) Division

– Includes development of specifications a

conformance tests for use by agencies anindustry

– Work is funded by Federal agencies and 

internal funds

• Homeland Security support of agencies 

investigating terrorist activities
Computer Forensics in ITL

• Goals of Computer Forensics Projects

– Introduce new automated processes into t

computer forensics investigative process

– Provide stable foundation built on scienti

rigor to support the introduction of evideand expert testimony in court
NSRL Software Collectio

• Media in format as available 

to the public

• Consumer products available 

in stores

• Developer products available 

as vendor services

• Malicious software• “Cracked” software
NSRL Software Collectio

• Balance of most popular (encountered 

often) and most desired (pirated often) 

– Currently 32 languages

• Software is purchased commercially• Software is donated under non-use pol• List of contents available on website

www.nsrl.nist.gov
NSRL Software Database

• Information to uniquely identify every 

on every piece of media in every appli

• Database schema is available on webs• 4,200 Bytes per application• 750 Bytes per file• Total database size now 9 GB for 3,80

applications with 13,400,000 files

NSRL Reference Data Se

• The Reference Data Set (RDS) is a sel

of information from the NSRL databas

• Allows positive identification of 

manufacturer, product, operating systeversion, file name from file “signature

• Data format available for forensic tool 

developers

• Published quarterly
Use of the RDS

• Eliminate as many known files as possible f

the examination process using automated m

• Discover expected file name with unknown 

contents

• Identify origins of files• Look for malicious files, e.g., hacker tools• Provide rigorously verified data for forensic 

investigations
RDS Field Use Example

Windows 2000 operating system software contain5933 images which are known gifs, icons, jpeg filYou are looking for facility maps on a computer which is running Windows 2000.By using the RDS and an analysis program the investigator would not have to look at these files tcomplete his investigation.e.g., 
RDS Field Use Concept

KNOWN FILES

RDS

ANALYSIS

PROGRAM

UNKNOWN FILES

FILES

Disk Drive
http://www.cbsnews.com/stories/2003/01/16/tech/main536774.shtml
Hashes

• Like a person’s fingerprint• Uniquely identifies the file based on conten• You can’t create the file from the hash• Primary hash value used is Secure Hash 

Algorithm (SHA-1) specified in FIPS 180-1160-bit hashing algorithm

– 1045 combinations of 160-bit values

• “Computationally infeasible” to find two di

files less than 264 bits in size producing the 

SHA-1

– 264 bits is one million terabytes
Hashes

• SHA-1 values can be cross-referenced 

other products that depend on differenvalues

• Other standard hash values computed 

each file include Message Digest 5 (Mand a 32-bit Cyclical Redundancy Checksum (CRC32), which are useful tools and to users outside LE
Hash Examples

Filename Bytes SHA-1

NT4\ALPHA\notepad.exe 68368 F1F284D5D757039DEC1C44A05AC1NT4\I386\notepad.exe 45328 3C4E15A29014358C61548A981A4ANT4\MIPS\notepad.exe 66832 33309956E4DBBA665E86962308FENT4\PPC\notepad.exe 68880 47BB7AF0E4DD565ED75DEB492D8CWINNT31.WKS\I386\notepad.exe 57252 2E0849CF327709FC46B705EEAB5EWINNT31.SRV\I386\notepad.exe 57252 2E0849CF327709FC46B705EEAB5E
Related History

• CRC concept dates from 1960’s• MD5 algorithm published in 1991• Tripwire open source tool 1992• Unix command “md5sum” available• FIPS 180-1 (SHA-1) published in 199• Unix command “sha1sum” available• Known File Filter project 1998• FIPS 180-2 (SHA-512) published in 2
Hashes in P2P

KaZaA Peer-to-Peer (P2P) FastTrack File Formatshttp://kzfti.cjb.net/
SHA-1 Mathematics

• Bit sequence is padded to a multiple of 512• Messages of 16 32-bit words, n*512, n>0• 80 logic functions are defined that accept 3 

words and produce 1 32-bit word

• 80 constants defined, 5 32-bit buffers initial• 80 step loop: 

– Manipulate message into 80 32-bit words– Use shifts, functions, addition on buffers

• 160-bit SHA is string in the 5 32-bit buffers
Application of RDS
NIST Research

• Hash collisions• Software distribution metrics• Operating/File system effects • Physical/Virtual machine effects• “Mining” dynamic files• Offsite hashing
Software Installation Issue

• Dynamic files are “missed” by RDS• Installed on virtual machines which ca

saved in the NSRL on media

• Delineation of static sections of files f

probability of identification

• Independent of installation location
NARA Research

• Use hashing process on non-classified 

Presidential materials

• Identify application files• Identify duplicate files• Access to older installed software
NARA Statistics

• 93 computer systems 

– Pre-filtered to contain only software

• 51,146 individual files • 7,610 file names • 11,118 distinct files (SHA-1)• 8,077 files originating in specific applicati• 4,326 of 8,077 exactly match application f

names
Further NARA Research

• Building profile of a “master” image• Statistical weights for application 

identification

• Cross-system relationships• Installation locations• Old compression technologies
NSRL Environment

• Isolated network with domain controller, DH• Database server, File server, Web server• Batching stations use web browser interface• Hashing constellation• Virtual machines for installations• CVS source code repository

Hashing Operations

• Spring 2003 – accepting software• Hashing constellation runs 24/7• Processed over 13.4M files, 9M SHAs• Byte signature file type verification• CAB, ZIP, TAR, SFX, UU, compress
Data Verification

• Multiple and independent techniques from d

perspectives

– We use test files with known signatures– Parallel database system: Match results with oth– Human verification– Database rules and constraints– Periodic database queries: Predefined procedure

search for and report anomalies in the database

– User feedback: Error reports and RDS updates
Future Operation Tasks

• More hardware platforms• More archive tools• Redundant hashing in constellation• Scheduled rebatching• Additional algorithms – AES• Open source LAMP distribution
NSRL/CFTT Team
Contact

NSRL Project, Douglas WhiteTelephone: 301-975-8425Email: nsrl@nist.govWeb: www.nsrl.nist.gov
Contacts

Jim Lyle Doug Whitewww.cftt.nist.gov www.nsrl.nist.gocftt@nist.gov nsrl@nist.govMark SkallChief, Software Diagnostics & Conformance Testing www.itl.nist.gov/div897 skall@nist.govSue Ballou, Office of Law Enforcement StandardsSteering Committee Rep. For State/Local Law Enforcsusan.ballou@nist.gov