8/28/2003 

NSRL Project NSRL Project
8/28/2003 

Introduction Introduction

The National Software Reference Library is:z A physical collection of over 4,000 software 

packages on secured shelves

z A database of file “fingerprints” and additio

information to uniquely identify each file on shelves

z A Reference Data Set (RDS) extracted from 

database onto CD, used by law enforcementinvestigators and researchers
8/28/2003 

Addressing Industry Nee

Addressing Industry Nee

z No unbiased organizations were involved in 

implementing investigative tools

z Law enforcement had no control over qualit

data provided by available tools – data was market-driven

z Traceability - No repositories of original sof

available for reproducing data

z Each tool provided a limited set of capabilit
8/28/2003 

NSRL Software Collectio

NSRL Software Collectio

z Media in format as available 

to the public

z Consumer products available 

in stores

z Developer products available 

as vendor services

z Malicious softwarez “Cracked” software
8/28/2003 

NSRL Software Collectio

NSRL Software Collectio

z Balance of most popular (encountered 

often) and most desired (pirated often) 

– Currently 32 languages

z Software is purchased commerciallyz Software is donated under non-use polz List of contents available on website

www.nsrl.nist.gov
8/28/2003 

NSRL Software Databas

NSRL Software Databas

z Information to uniquely identify every 

on every piece of media in every appli

z Database schema is available on websz 4,200 Bytes per applicationz 750 Bytes per filez Total database size is 11 GB for 4,000 

applications with 15,000,000 files
8/28/2003
8/28/2003 

NSRL Reference Data S

NSRL Reference Data S

z The Reference Data Set (RDS) is a sel

of information from the NSRL databas

z Allows positive identification of 

manufacturer, product, operating systeversion, file name from file “signature

z Data format available for forensic tool 

developers

z Published quarterly
8/28/2003 

Use of the RDS Use of the RDS

z Eliminate as many known files as possible f

the examination process using automated m

z Discover expected file name with unknown 

contents

z Identify origins of filesz Look for malicious files, e.g., hacker toolsz Provide rigorously verified data for forensic 

investigations
8/28/2003 

RDS Field Use Example RDS Field Use Example

Windows 2000 operating system software contain5933 images which are known gifs, icons, jpeg filYou are looking for facility maps on a computer which is running Windows 2000.By using the RDS and an analysis program the investigator would not have to look at these files tcomplete her investigation.e.g., 
8/28/2003 

RDS Field Use Concep

RDS Field Use Concep

KNOWN FILES

RDS

ANALYSIS

PROGRAM

UNKNOWN FILE

FILES

Disk Drive
8/28/2003 

http://www.cbsnews.com/stories/2003/01/16/tech/main536774.shtml
8/28/2003 

Hashes

z Like a person’s fingerprintz Uniquely identifies the file based on contenz You can’t create the file from the hashz Primary hash value used is Secure Hash 

Algorithm (SHA-1) specified in FIPS 180-1160-bit hashing algorithm

– 1045 combinations of 160-bit values

z “Computationally infeasible” to find two di

files less than 264 bits in size producing the 

SHA-1

– 264 bits is one million terabytes
8/28/2003 

Hashes

z SHA-1 values can be cross-referenced 

other products that depend on differenvalues

z Other standard hash values computed 

each file include Message Digest 5 (Mand a 32-bit Cyclical Redundancy Checksum (CRC32), which are useful tools and to users outside LE
8/28/2003 

Hash Examples Hash Examples

Filename Bytes SHA-1

NT4\ALPHA\notepad.exe 68368 F1F284D5D757039DEC1C44A05AC1NT4\I386\notepad.exe 45328 3C4E15A29014358C61548A981A4ANT4\MIPS\notepad.exe 66832 33309956E4DBBA665E86962308FENT4\PPC\notepad.exe 68880 47BB7AF0E4DD565ED75DEB492D8CWINNT31.WKS\I386\notepad.exe 57252 2E0849CF327709FC46B705EEAB5WINNT31.SRV\I386\notepad.exe 57252 2E0849CF327709FC46B705EEAB5Filename CRC32 MD5 Bytes SHA-1null.dat 0 DA39A3EE5E6B4B0D3255BFEF956

00000000 D41D8CD98F00B204E9800998ECF8427E
8/28/2003 

Related History Related History

z CRC concept dates from 1960’sz MD5 algorithm published in 1991z Tripwire open source tool 1992z Unix command “md5sum” availablez FIPS 180-1 (SHA-1) published in 199z Unix command “sha1sum” availablez Known File Filter project 1998z FIPS 180-2 (SHA-512) published in 2
8/28/2003 

Hashes in P2P Hashes in P2P

KaZaA Peer-to-Peer (P2P) FastTrack File Formatshttp://kzfti.cjb.net/
8/28/2003 

SHA-1 Mathematics 1 Mathematics

z Bit sequence is padded to a multiple of 512z Messages of 16 32-bit words, n*512, n>0z 80 logic functions are defined that accept 3 

words and produce 1 32-bit word

z 80 constants defined, 5 32-bit buffers initialz 80 step loop: 

– Manipulate message into 80 32-bit words– Use shifts, functions, addition on buffers

z 160-bit SHA is string in the 5 32-bit buffers
8/28/2003 

Effectiveness of RDS Effectiveness of RDS
8/28/2003 

Hashkeeper Hashkeeper Compariso

Compariso

z May 2002 article by Dan Mares comparing Hashk

NSRL

z http://www.scmagazine.com/scmagazine/sconline/2002/article/24/article.htmlz http://www.nsrl.nist.gov/documents/dm_july02/z Using Hashkeeper 001-243 and NSRL 1.2 (June 2Source Unique MD5s MD5s in Hashkeeper MD5s in NSRL Comm

in data file NOT in NSRL NOT in Hashkeeper to B

NSRL 4,022,258 3,777,082 245,Hashkeeper 766,854 411,962 245,
8/28/2003 

NIST Research NIST Research

z Hash collisionsz Software distribution metricsz Operating/File system effects z Physical/Virtual machine effectsz “Mining” dynamic filesz Offsite hashing
8/28/2003 

Software Installation Issu

Software Installation Issu

z Dynamic files are “missed” by RDSz Installed on virtual machines which ca

saved in the NSRL on media

z Delineation of static sections of files f

probability of identification

z Independent of installation location
8/28/2003 

NARA Research NARA Research

z Use hashing process on non-classified 

Presidential materials

z Identify application filesz Identify duplicate filesz Access to older installed software
8/28/2003 

NARA Statistics NARA Statistics

z 93 computer systems 

– Pre-filtered to contain only software

z 51,146 individual files z 7,610 file names z 11,118 distinct files (SHA-1)z 8,077 files originating in specific applicatiz 4,326 of 8,077 exactly match application f

names
8/28/2003 

Further NARA Researc

Further NARA Researc

z Building profile of a “master” imagez Statistical weights for application 

identification

z Cross-system relationshipsz Installation locationsz Old compression technologies
8/28/2003 

NSRL Environment NSRL Environment

z Isolated network with domain controller, DHz Database server, File server, Web serverz Batching stations use web browser interfacez Hashing constellationz Virtual machines for installationsz CVS source code repository
8/28/2003 

Input Process Input Process

z Package is acquiredz Web interface used to enter informatio

about manufacturer, product, OS and aan ID

z Media are batchedz Approximately 15 minutes per packag
8/28/2003 
8/28/2003 

Hashing Operations Hashing Operations

z Spring 2003 – accepting softwarez Hashing constellation runs 24/7z Processed over 15M files, 10M SHAsz Byte signature file type verificationz CAB, ZIP, TAR, SFX, UU, compress
8/28/2003 

Hash Calculation Times Hash Calculation Times

Statistics on three runs totalling 10GB of data

User+System Time = 740.5350 Seconds

%Time ExclSec CumulS #Calls sec/call Name

41.2 305.2 304.69 1000 0.0031 Digest::SHA1::addfile28.5 211.1 210.58 1000 0.0021 String::CRC32::crc3223.8 176.3 175.75 1000 0.0018 Digest::MD5::addfile

User+System Time = 791.8629 Seconds

%Time ExclSec CumulS #Calls sec/call Name

42.8 339.6 339.64 100 3.3965 Digest::SHA1::addfile30.3 240.6 240.64 100 2.4065 String::CRC32::crc3226.6 211.2 211.25 100 2.1126 Digest::MD5::addfile

User+System Time = 836.9632 Seconds

%Time ExclSec CumulS #Calls sec/call Name

42.4 355.1 355.12 10 35.512 Digest::SHA1::addfile30.5 255.3 255.31 10 25.531 String::CRC32::crc3227.0 226.4 226.41 10 22.641 Digest::MD5::addfile
8/28/2003 

Data Verification Data Verification

z Multiple and independent techniques from d

perspectives

– We use test files with known signatures– Parallel database system: Match results with oth– Human verification– Database rules and constraints– Periodic database queries: Predefined procedure

search for and report anomalies in the database

– User feedback: Error reports and RDS updates
8/28/2003 

Future Operation Tasks Future Operation Tasks

z More hardware platformsz More archive toolsz Redundant hashing in constellationz Scheduled rebatchingz Additional algorithms – AESz Open source LAMP distribution
8/28/2003 

NSRL Accomplishment

NSRL Accomplishment

z RDS CD Version 1.5 distributed 3/3/2

– 102 subscriptions (Vendors, corporations

universities, and law enforcement agenci

– Free redistribution, NIST traceable

z Incorporated into vendor productsz Used by FBI, DCCC, Secret Service, 

Customs Service (Homeland Security)
8/28/2003 

NSRL/CFTT Team NSRL/CFTT Team
8/28/2003 

Contacts Contacts

Jim Lyle Doug Whitewww.cftt.nist.gov www.nsrl.nist.gocftt@nist.gov nsrl@nist.govBarbara Guttmanbarbara.guttman@nist.govSue Ballou, Office of Law Enforcement StandardsSteering Committee Rep. For State/Local Law Enforcsusan.ballou@nist.gov 
8/28/2003