Identification of Known Files

on Computer Systems

AAFS 2005

Douglas White

Michael Ogata
Disclaimer

Trade names and company products are

mentioned in the text or identified. In no

case does such identification imply

recommendation or endorsement by the

National Institute of Standards and

Technology, nor does it imply that the

products are necessarily the best available

for the purpose.
Source: FBI CART Team, Oct. 2003

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

FY '99 FY '00 FY '01 FY '02 FY '03

Case Data

FBI’s Cyber Caseload and Dataset Size Growth

 Source: FBI CART, Oct 2003

Problem: Data Inflation
NIST Digital Forensics Goals

 Provide standard reference data that

investigators and tool makers can use

 Assist in reducing manual processes in case

loads, reducing case processing time

 Identify known files, allowing investigator to

focus on user-generated data
Known File Identification

Digital fingerprint, or “hash”

 Cryptographic function: MD5, SHA-1

 Like human fingerprint, can’t rebuild

original from this information

 Extremely hard to circumvent

– Be aware of collision research
Related History

 CRC concept dates from 1960’s

 MD5 algorithm published in 1991

 Tripwire open source tool 1992

 Hash command “md5sum” available

 FIPS 180-1 (SHA-1) published in 1995

 Hash command “sha1sum” available

 Known File Filter project 1998

 FIPS 180-2 (SHA-512) published in 2002

 Hash command “sha2sum” available
Hash Examples

 Filename Bytes SHA-1

NT4\ALPHA\notepad.exe 68368 F1F284D5D757039DEC1C44A05AC148B9D204E467

NT4\I386\notepad.exe 45328 3C4E15A29014358C61548A981A4AC8573167BE37

NT4\MIPS\notepad.exe 66832 33309956E4DBBA665E86962308FE5E1378998E69

NT4\PPC\notepad.exe 68880 47BB7AF0E4DD565ED75DEB492D8C17B1BFD3FB23

NT31WS\I386\notepad.exe 57252 2E0849CF327709FC46B705EEAB5E57380F5B1F67

NT31SRV\I386\notepad.exe 57252 2E0849CF327709FC46B705EEAB5E57380F5B1F67

contract.txt 0BD71F653A5B83E61D66DB6D29B9B46655D77F42
Hash Application

Which was the original?

contract1.txt

John Doe owes Rachel Roe $15.00

contract2.txt

John Doe owes Rachel Roe $1500.
Hash Application

sha1sum contract*

0BD71F653A5B83E61D66DB6D29B9B46655D77F42 contract1.txt

B10A4DEDC819737E7D62363ADE0A2F035A2CC20F contract2.txt

0BD71F653A5B83E61D66DB6D29B9B46655D77F42 contract.txt
Hashset Sources

 NIST NSRL

 NDIC HashKeeper

 Maresware

 Tripwire FSDB

 Known Goods website

 Vendors, e.g. Sun Solaris Fingerprints

 CFTT, iLook, CFID email lists

 Professional connections
EnCase Screen
EnCase Screen
FTK screen
FTK screen
Identification Metrics

Win ME 5,169 93% 383 11,512

Win XPPro 9,404 86% 1,293 19,546

Win 2KPro 5,963 86% 839 16,539

Win NT 1,659 86% 239 17,904

WS

Win 98 4,266 93% 297 18,662

Files in

Distribution

Files

Unknown

Percent

Identified

Files

Installed

Operating

System

Compare hashes from known OS media to hashes of

installation of that OS; best case scenario
Identification Metrics

Files in

Distribution

Files

Unknown

Percent

Identified

Files

Installed

Operating

System

Win ME + 24,112 98% 526 32,758

Office 2K

Win 98 + 23,464 98% 596 43,327

Office 2K

Compare hashes from known media to hashes of

installations; best case scenario
Identification in Practice

NIST PC

#5 W98

NIST PC

#4 W98

NIST PC

#3 WNT

NIST PC

#1 W2K

NIST PC

#2 W2K

Operating

System

34,220

16,397

14,186

18,048

59,135

Files

Installed

75%

55%

54%

35%

20%

Percent

Identified

8,667

7,404

6,618

11,839

47,124

Files

Unknown

Project

development

Researcher,

Several apps

Researcher,

Several apps

“Normal” use

Email, writing

Manager’s PC

email, memos

Notes
File Identification on a Changing

Windows 2000 System
Hashing Limitations

 Eliminate known files on seized machine

 Only as good as the hashed collection

 Applicable feedback from installations

 Dynamic files - may use block size hashes

 Audio, images easily changed
NARA Research

 Use hashing process on non-classified

Presidential materials

 Identify application files

 Identify duplicate files

 Access to older installed software
NARA Statistics

 93 computer systems

– Pre-filtered to contain only software

 51,146 individual files

 11,118 distinct files (SHA-1)

 8,077 files originating in specific application(s)

 7,610 file names

 4,326 of 8,077 exactly match application file names

 Able to trace system “pedigree”
Contacts

Doug White

Michael Ogata

www.nsrl.nist.gov

nsrl@nist.gov

Barbara Guttman

barbara.guttman@nist.gov

Sue Ballou, Office of Law Enforcement Standards

Rep. For State/Local Law Enforcement

susan.ballou@nist.gov

NSRL Software Collection

 Media in format as available to

the public

 Consumer products available

in stores

 Developer products available

as vendor services

 Malicious software

 “Cracked” software
Hash Verification

www.nsrl.nist.gov/testdata
Hash Collision News

 The NSRL project does not see any fatal ramifications from the collision

announcements.

 Details posted at http://www.nsrl.nist.gov/collision.html within 2 days

 This was not a "pre-image" attack; that is, the researchers did not identify a

known file in the NSRL and attempt to generate a different file with a matching

hash value.

 Nothing presented at Crypto 2004 indicated that SHA-1 has been broken

 There are known MD5 collisions and weaknesses; the NSRL data provides an

MD5 to SHA-1 mapping to facilitate the migration away from MD5.

 SHA-1 will be superceded in 2010 by FIPS 180-2, Secure Hash Standard

(SHA-224, 256, 384,512). The NSRL will provide a SHA-1 to SHA-256 mapping.

 The NSRL provides several hash values and the file size, and it is highly

improbable that a pre-image attack will be found soon that can generate a

combination of hash collisions.
Hashes

 Like a person’s fingerprint

 Uniquely identifies the file based on contents

 You can’t create the file from the hash

 Primary hash value used is Secure Hash Algorithm

(SHA-1) specified in FIPS 180-1, a 160-bit hashing

algorithm

– 1045 combinations of 160-bit values

 “Computationally infeasible” to find two different files

less than 264 bits in size producing the same SHA-1

– 264 bits is one million terabytes
SHA-1 Mathematics

 Bit sequence is padded to a multiple of 512

 Messages of 16 32-bit words, n*512, n>0

 80 logic functions are defined that accept 3 32-bit

words and produce 1 32-bit word

 80 constants defined, 5 32-bit buffers initialized

 80 step loop:

– Manipulate message into 80 32-bit words

– Use shifts, functions, addition on buffers

 160-bit SHA is string in the 5 32-bit buffers