Computer Forensics Tool 

Testing

Jim Lyle

National Institute of Standards 

and Technology
November 30, 2012 Forensics@NIST 2012 2

Disclaimer

 Certain trade names and company 

products are mentioned in the text or 

identified. In no case does such 

identification imply recommendation or 

endorsement by the National Institute of 

Standards and Technology, nor does it 

imply that the products are necessarily 

the best available for the purpose.
November 30, 2012 Forensics@NIST 2012 4

Overview

Mostly high level thoughts and a few details 

about testing at CFTT

Conformance testing as used by CFTT

Some challenges for writing requirements 

and test cases

Selecting test cases

Thoughts on testing acquisition tools, write 

blocker tools and disk wiping tools
Current Tasks

Testing tools for: 

imaging, write blocking, drive wipe

deleted file recovery, file carving 

mobile devices

New specs for: tablet devices, File 

Carving

Forensic tool catalog

Federated testing

November 30, 2012 Forensics@NIST 2012 5
November 30, 2012 Forensics@NIST 2012 6

General CFTT Method:

Conformance Testing

Requirements specification

Background & definitions

Core behaviors for all tools

Optional features and behaviors

Test Assertions -- atomic tests

Test Cases -- each case evaluates some 

subset of the assertions

Test procedures -- how to run tests

Test Reports
November 30, 2012 Forensics@NIST 2012 7

CFTT Reports

(since Aug 2002)

Tool Type Published Testing/Drafting Report

Disk imaging 21 7

Software Blocker 9 0

Hardware Blocker 24 0

Mobile Devices 21 5

Drive erase/wipe 10 0

Deleted File Recovery 0 6

Total as of Nov 2012 85 18
November 30, 2012 Forensics@NIST 2012 8

Challenges to Creating a 

Specification

 Diversity of tool features 

 May not be one correct behavior

Write blocker behavior

Deleted file recovery

File Carving

 Some actions not exactly repeatable, e.g., memory 

acquire

 Needs to allow for evolution of technology
November 30, 2012 Forensics@NIST 2012 9

Challenges to Creating Test 

Cases

Many errors only manifest if there is a specific 

set of conditions.

Combinatorics -- testing enough 

combinations of parameters & possible 

values

Example (creating a disk image)

Partition: FAT, NTFS, ext3, HFS

Physical: ATA, SCSI, USB, 1394

Destination: image, clone

Error: none, bad sector, out of space

4x4x2x3 = 96 runs -- at 3 hours/run -- 288 hours

or 36 days or about 7 weeks
Federated Testing

NIST develops specification, methods & 

test data

Forensic labs do tool testing

Test reports follow standardized format

Reviewed Test reports posted to central 

web site

November 30, 2012 Forensics@NIST 2012 10
November 30, 2012 Forensics@NIST 2012 11
November 30, 2012 Forensics@NIST 2012 12
November 30, 2012 Forensics@NIST 2012 13
CFReDS:Forensic Tool Test 

Data

 Data sets for tool testing

 Deleted File Recovery

 File Carving

 String Search UNICODE text in русский

 Search container files (e.g., zip or tar)

 Memory images

 Mobile device images

 Data sets for equipment check out

 Data sets for staff training

 Proficiency Testing and Skill Testing

November 30, 2012 Forensics@NIST 2012 14
Testing Acquisition Tools

Primary issues

Clone a device

Acquire a device to an image file

Restore image file to a device 

Secondary issues

Partitions

Hidden areas

Reliably faulty drives

November 30, 2012 Forensics@NIST 2012 17
November 30, 2012 Forensics@NIST 2012 18

Testing Write Blockers

Use cmd generator to send all possible I/O 

commands (even undefined commands)

Monitor blocker output to characterize tool 

behavior (preferred measurement method)

All writes must be blocked

At least one read cmd must be allowed

Just report on behavior for anything else

Alternate test cases (using different 

measurements) if can’t use generator or 

monitor
November 30, 2012 Forensics@NIST 2012 19

Forensic Media Preparation

Disk wiping for internal reuse (not for 

disposal)

For disposal see: NIST SP 800-88 

Guidelines for Media Sanitization: 

Recommendations of the National 

Institute of Standards and Technology

Write vs SECURE ERASE
DFR Testing Overview

 About 17 test cases defined (1 run for each file 

system family, 4 runs per case)

 Is particular file system supported

 Can active files be listed

 Support for non-ASCII file names

 Can deleted file names be recovered (maybe not)

 Recover contiguous content

 Recover fragmented content

 Identify overwritten content

November 30, 2012 Forensics@NIST 2012 23
Testing

Supported File Systems

 Basic Case to identify supported file systems

 Test case –

1. Create three files: A, B & C

2. Delete file B

3. Image file system

OS File Systems

WIN FAT 12/16/32 NT NTC ExFAT

Mac HFS OSX OSX-J OSX-C OSX-JC

Linux Ext2 Ext3 Ext4

November 30, 2012 Forensics@NIST 2012 24
Supported File Systems: 

Results

Tool Active File List Deleted File 

Name

Deleted File 

Content

Tool #1 FAT,HFS,OSX, 

OSXJ, EXT

Nothing: OSXC, 

OSXCJ

FAT, NT

No name: OSX, 

EXT

FAT, NT, ext2 (as 

lost file), 

Nothing: ext3/4, 

HFS, 

OSX/C/J 

Tool #2 FAT, NT, HFS, 

OSX/C/J, 

EXT2/3 Nothing: 

ext4

FAT, NT

No name: OSX, 

EXT

FAT, NT, EXT2

No content: 

NTC, OSX/J/C, 

EXT3/4

Tool #3 FAT, NT, HFS, 

OSX/C/J, EXT

Nothing: OSXCJ

FAT, NT

No name: OSX, 

EXT

FAT, NT, EXT2

No content: 

OXS/J/C, 

EXT3/4

November 30, 2012 Forensics@NIST 2012 25
Non-ASCII File Names

1. Create set of files with non-ASCII file 

names

 European diacritical marks

 Asian characters

 Right to left text

2. Delete some files

3. Run tools (#1, #2 & #3)

November 30, 2012 Forensics@NIST 2012 26
Non-ASCII File Names

• Most tools rendered non-ASCII correctly for most file 

systems.

•Two tools had problem rendering Korean text from OSX

•One tool could not render non-ASCII file names from EXT2

November 30, 2012 Forensics@NIST 2012 27
Mobile Device Tools

Stick around for Rick (after the break)

November 30, 2012 Forensics@NIST 2012 28
November 30, 2012 Forensics@NIST 2012 29

Summary

 Give tool opportunity to fail -- diverse test suite & fault based test 

cases

 Case templates that vary over a parameter -- this is useful as 

technology evolves

 Use pair-wise testing to allocate lots of parameters among a few 

test cases

 Have alternate cases with different measurement tools if first 

measurement method can’t be used

 Make tool test data available to community via 

www.cfreds.nist.gov

 Make test methodology available to community via Federated 

Testing Project 
November 30, 2012 Forensics@NIST 2012 30

Contacts

Jim Lyle Doug White

www.cftt.nist.gov www.nsrl.nist.gov

www.cfreds.nist.gov

cftt@nist.gov nsrl@nist.gov

Sue Ballou, Office of Law Enforcement Standards

Steering Committee Rep. For State/Local Law Enforcement

susan.ballou@nist.gov