NSRL Project NSRL Project
Outline Outline

z NSRL Descriptionz RDS Descriptionz RDS Field Usez Hash System Overview
What is the NSRL? What is the NSRL?

z National Software Reference Library (NSRL)

– Physical library of software, 1900 products– SQL Server database of known file signatures– Reference Data Set (RDS)

z Extract of database on CD: 6,500,000 file signatures

z Goals

– Automate the process of identifying known files 

computers used in crimes

– Allow investigators to concentrate on files that c

contain evidence (unknown and suspect files)
Addressing Law Addressing Law 

Enforcement Needs Enforcement Needs

z LE needed an unbiased organizationz LE needed traceability for the NSRL 

contents

z No repositories of original software 

available for reproducing data

z NSRL needs to work with many CF to
Scope of the NSRL Scope of the NSRL

z NIST has collected software for 2 yz Software is recorded as the origina

for known files and stored as a part NSRL

z Versions of OS, DBMS, photo edit

word processors, network browsercompilers...

z Data formats, data dictionary and p

status information is available on thwebsite for RDS users and industry reference
What is the RDS? What is the RDS?
What is the RDS? What is the RDS?

z Reference set of file profiles

– Each profile includes file name, file size, 

signatures (SHA1, MD4, MD5, CRC32), application name, operating system, etc.

– Extracted from files on original software 

diskettes, and network downloads

– A single application may have 10,000 sep

file profiles
How to Use the RDS How to Use the RDS

z Eliminate as many known files as possible from the 

examination process using automated means

z Discover files that do not contain expected content

file containing a bomb schematic, facility map)

z Look for files that should be installed, but are miss

(incomplete deletion of pirated software)

z Look for files that could be suspect (hash matches, 

name does not)

z Provide rigorously verified data for forensic investi
RDS Field Use RDS Field Use

KNOWN FILES

RDS

ANALYSIS

PROGRAM

UNKNOWN FILES

FILES

40-95%

Disk Drive
RDS Field Use Example RDS Field Use Exampl

Windows NT 4.0 operating system software conta6753 images which are known gifs, icons, jpeg filYou are looking for facility maps on a computer which is running Windows NT 4.0 Workstation.By using the RDS and an analysis program the investigator would not have to look at these files tcomplete his investigation.

e.g., 
Hash System Overview Hash System Overview

z Environmentz Input Processz Hashingz Verificationz Future
Environment Environment

z Isolated LANz Domain controllerz Database serverz File serverz CVS repositoryz Virtual Machinesz Batching stationsz Hashing constellation
General Process General Process
Input Process Input Process

z Package is acquiredz Web interface used to enter informatio

about manufacturer, product, OS and aan ID

z Media are batchedz Approximately 15 minutes per packag

Hashing Process Hashing Process

z July 2002 – accepting software, perfor

installations

z Currently hashing 5 p.m. until donez 6 200MHz PCs in hashing constellatioz Averaging 10.5 hashes/sec.z DVD with 250,000 files needed 30 ho
Hashes

z Compute a unique identifier for each file ba

contents 

z Primary hash value used in the NSRL RDS 

Secure Hash Algorithm (SHA-1) specified iFederal Information Processing Standard (F180-1, a 160-bit hashing algorithm

z SHA-1 values can be cross-referenced by ot

products that depend on different hash value
Hashes

z Other standard hash values computed for each file 

Message Digest 4 (MD4), Message Digest 5 (MD532-bit Cyclical Redundancy Checksum (CRC32), ware useful in many CF tools and to users outside LE

z Separate, parallel, and independent process is used 

validate the results of the primary RDS implementa

z Once verified and validated, the RDS is written to a 

CD, duplicated, and distributed through NIST’s StaReference Data Office as Special Database #28 (www.nist.gov/srd/nistsd28.htm).
Hash Examples Hash Examples

Filename Bytes SHA-1

NT4\ALPHA\notepad.exe 68368 F1F284D5D757039DEC1C44A05AC1NT4\I386\notepad.exe 45328 3C4E15A29014358C61548A981A4ANT4\MIPS\notepad.exe 66832 33309956E4DBBA665E86962308FENT4\PPC\notepad.exe 68880 47BB7AF0E4DD565ED75DEB492D8CWINNT31.WKS\I386\notepad.exe 57252 2E0849CF327709FC46B705EEAB5EWINNT31.SRV\I386\notepad.exe 57252 2E0849CF327709FC46B705EEAB5E
Installation Hashes Installation Hashes

z 300-800 files are “missed” by current z Compare automated hashes with real-w

installed hashes

z Compare installed file sets across mac

and OS’es

z Installed on virtual machines which ca

saved in the NSRL on media
Use on Actual Machine

Use on Actual Machine

Clean OSz 4622 files -360, 92% known – w98z 7720 files -864, 89% known – w2kz 5412 files -370, 93% known – wme

Actual NIST PCs

z 39631 files -7902, 80% known – w2kz 18262 files -6395, 65% known – w98z 75834 files -41638, 45% known – w2k,mgm

Data Verification Data Verification

z Multiple and independent techniques from d

perspectives

– We use test files with known signatures– Parallel database system: Match results with oth– Human verification– Database rules and constraints– Periodic database queries: Predefined procedure

search for and report anomalies in the database

– User feedback: Error reports and RDS updates
Future Tasks Future Tasks

z Byte signature file type verificationz Self-extracting EXE filesz Redundant hashing in constellationz Scheduled rebatchingz Additional algorithms - AES
NSRL Accomplishment

NSRL Accomplishment

z RDS CD Version 1.2 distributed 6/6/2

– 124 subscriptions (Vendors, corporations

universities, and law enforcement agenci

– Free redistribution, NIST traceable

z Incorporated into vendor productsz Used by FBI, DCCC, Secret Service, 

Customs Service (Homeland Security)
CFTT/NSRL Team CFTT/NSRL Team
Contacts Contacts

Jim Lyle Doug Whitewww.cftt.nist.gov www.nsrl.nist.gocftt@nist.gov nsrl@nist.govMark SkallChief, Software Diagnostics & Conformance Testing www.itl.nist.gov/div897 skall@nist.govSue Ballou, Office of Law Enforcement StandardsSteering Committee Rep. For State/Local Law Enforcsusan.ballou@nist.gov