Common-Mode Failures:
What Can You Do With 236 750 Used Hard Drives?

Simson L. Garfinkel
Postdoctoral fellow, Center for Research on Computation and Society
Harvard University
December 7, 2005

1

Purchased used from a computer store in August 1998:

2

Computer #1: 486-class machine with 32MB of RAM

A law firm’s file server...
...with client documents!

Computers #2 through #10 had:
• Mental health records• Home finances• Draft of a novel...
Was this a chance accident or common occurrence?
3This talk presents the disk sanitization problem anddiscusses a new technique for computer forensics.
2, 500

Megabytes

2, 0001, 500No Data (blocks cleared)
Data not in the file system (level 2 and 3)
Data in the file system (level 0)

1, 000500

1. Scale of the problem

0

2. The Traceback Study
many of these sources, their credibility was difficult to assess and was often left to the foreign
government services to judge. Intelligence Community HUMINT efforts against a closed society
like Iraq prior to Operation Iraqi Freedom were hobbled by the Intelligence Community's
dependence on having an official U.S. presence in-country to mount clandestine HUMINT
collection efforts.
(U) When UN inspectors departed Iraq, the placement of HUMINT agents and thedevelopment of unilateral sources inside Iraq were not top priorities for the Intelligence
Community. The Intelligence Community did not have a single HUMINT source collecting
against Iraq's weapons of mass destruction programs in Iraq after 1998. The Intelligence
Community appears to have decided that the difficulty and risks inherent in developing sources
or inserting operations officers into Iraq outweighed the potential benefits. The Committee
found no evidence that a lack of resources significantly prevented the Intelligence Communityfrom developing sources or inserting operations officers into Iraq.

When Committee staff asked why the CIA had not considered
placing a CIA officer in Iraq years before Operation Iraqi Freedom to investigate Iraq's weaponsof mass destruction programs, a CIA officer said, "because it's very hard to sustain . . . it takes arare officer who can go in . . . and survive scrutiny | ^ | [ m | | | for a long time." TheCommittee agrees that such operations are difficult and dangerous, but they should be within thenorm of the CIA's activities and capabilities. Senior CIA officials have repeatedly told theCommittee that a significant increase in funding and personnel will be required to enable to the
CIA to penetrate difficult HUMINT targets similar to prewar Iraq. The Committee believes,
however, that if an officer willing and able to take such an assignment really is "rare" at the CIA,
the problem is less a question of resources than a need for dramatic changes in a risk averse
corporate culture.
(U) Problems with the Intelligence Community's HUMINT efforts were also evident in
the Intelligence Community's handling of Iraq's alleged efforts to acquire uranium from Niger.
The Committee does not fault the CIA for exploiting the access enjoyed by the spouse of a CIA
employee traveling to Niger. The Committee believes, however, that it is unfortunate,considering the significant resources available to the CIA, that this was the only option available.
Given the nature of rapidly evolving global threats such as terrorism and the proliferation of
weapons and weapons technology, the Intelligence Community must develop means to quickly
respond to fleeting collection opportunities outside the Community's established operating areas.
The Committee also found other problems with the Intelligence Community's follow-up on the-25-

3. Common failures and solutions

4

Hard drives pose special problem for computer security

Do not forget data when power is removed.
Contain data that is not immediately visible.
Today’s computers can read hard drives thatare 15 years old!
• Electrically compatible (IDE/ATA)• Logically compatible(FAT16/32 file systems)• Very different from tape systems

5

Scale of the problem: huge!
400M
350M

Drives Shipped
Drives Retired

300M
250M
200M
150M
100M
50M
1996

1998

2000

2002

2004

2006

210 million drives will be retired this year.
6Physical destruction will remove the information...

...but many “retired” drives are not physically destroyed.
7

There is a significant secondary market for used disk drives.

Retired drives are:• Re-used withinorganizations• Given to charities• Sold at auction

About 1000 used drives/day sold on eBay.
8

Since January 1999,
I have acquired 750 hard drives on the secondary market.

9Drives arrive by UPS

10

Data on drives “imaged” using FreeBSD

11

Images stored on external firewire drives

This is 900GB of storage.

12For every drive, I catalog:• Disk SN, date of manufacture, etc.
• Every readable sector on the drive..
• All visible files.
• MD5 of every file.
• MD5 of the image.

13

Example: Disk #70: IBM-DALA-3540/81B70E32
Purchased for $5 from a Mass retail store on eBay
Copied the data off: 541MB
Initial analysis:
Total disk sectors:
1,057,392
Total non-zero sectors:
989,514
Total files:
3
The files:
drwxrwxrwx-r-xr-xr-x-r-xr-xr-x-rwxrwxrwx

0 root0 root0 root0 root0 Dec 31222390 May 11
9 May 1193880 May 11

1979 ./
1998 IO.SYS
1998 MSDOS.SYS
1998 COMMAND.COM
14

Clearly, this disk had been FORMATed...

Windows FORMAT doesn’t erase the disk...
FORMAT just writes a new root directory.

15

UNIX “strings” reveals the disk’s previous contents...

Insert diskette for drive
and press any key when readyYour program caused a divide overflow error.
If the problem persists, contact your program vendor.
Windows has disabled direct disk access to protect your lo
To override this protection, see the LOCK /? command for m
The system has been halted. Press Ctrl+Alt+Del to restart
You started your computer with a version of MS-DOS incompa
version of Windows. Insert a Startup diskette matching thi

OEMString = "NCR 14 inch Analog Color Display Enchanced SV
Graphics Mode: 640 x 480 at 72Hz vertical refresh.
XResolution= 640
YResolution= 480
VerticalRefresh= 72
16

70.img con’t...

ling the Trial Edition---------------------------IBM AntiVirus Trial Edition is a full-function but time-li
evaluation version of the IBM AntiVirus Desktop Edition pr
may have received the Trial Edition on a promotional CD-RO
single-file installation program over a network. The Tria
is available in seven national languages, and each languag
provided on a separate CC-ROM or as a separa
EAS.STCm
EET.STC
ELR.STCq
ELS.STC17

70.img con’t...
MAB-DEDUCTIBLE
MAB-MOOP
MAB-MOOP-DED
METHIMAZOLEINSULIN (HUMAN)
COUMARIN ANTICOAGULANTSCARBAMATE DERIVATIVESAMANTADINEMANNITOL
MAPROTILINECARBAMAZEPINECHLORPHENESIN CARBAMATEETHINAMATEFORMALDEHYDE
MAFENIDE ACETATE
18

[Garfinkel & Shelat 03] established the scale of the problem.

We found:• Thousands of credit card numbers• Financial records• Medical information• Trade secrets• Highly personal information

We did not determine why the data had been left behind.
19

There are roughly a dozen documented cases of people
purchasing old PCs and finding sensitive data.
• A woman in Pahrump, NV bought aused PC with pharmacy records[Markoff 97]• Pennsylvania sold PCs with“thousands of files” on state
employees [Villano 02]• Paul McCartney’s bank records soldby his bank [Leyden 04]• O&O Software GmbH – 200
drives.[O&O 05]

None of these cases are scientifically rigorous.
20Why don’t we hear more stories?
Hypothesis #1: Disclosure of “data passed” is exceedingly
rare because most systems are properly
cleared.
Hypothesis #2: Disclosures are so common that they are notnewsworthy.
Hypothesis #3: Systems aren’t properly cleared, but few
people notice the data.

21

How could people not notice the data?

DEL removes the file’s name; doesn’t delete the data.
22

FORMAT writes a new root directory and FAT.

FORMAT doesn’t doesn’t overwrite the disk sectors.

23

Data left behind on hard drives is a serious social problem.

Large numbers of drives are being soldand given away.
Many appear to have hidden
confidential information.

We are morally obligated to solve this problem!
24

[Garfinkel ’05] presents five distinct patterns
for addressing the sanitization problem
Visibility

Sanitization





Users

Users

User
Audit

Explicit Item
DeleteReset toInstallation

DelayedUnrecoverable
ActionComplete
DeleteDocument Files, Applications, and Media

http://www.simson.net/thesis/
25

To be effective, a solution must address the root cause

Usability Problem:

Education Problem:• Effective audit of information
present on drives.

• Add training to the interface.
[Whitten 04]• Make DEL and FORMAT
actually remove data.
[Bauer & Priyantha 01]• Regulatory requirements.
[FTC 05, SEC 05]• Legal liability.

• Provide alternative strategies fordata recovery.

To find that cause,
I looked on the drives and contacted the data subjects.
26

Data on a hard drive is arranged in sectors.

/

a

tmp

usr

bin

b

slg

ls

beth

mail

junk

cp

The white sectors indicate directories and files that are
visible to the user.

mv

27

Data on a hard drive is arranged in sectors.

x8

a

x7

/

x5

tmp

usr

x4

bin

x1

x6

b

slg

ls

cp

mv

beth

mail

junk

x3

x2

The brown sectors indicate files that were deleted.
28

Data on a hard drive is arranged in sectors.

x8

a

x7

/

x5

tmp

usr

x4

bin

x1

x6

b

slg

ls

cp

mv

beth

mail

junk

x3

x2

The green sectors indicate sectors that were never used (orthat were wiped clean).
29

Stack the disk sectors:

Zero Blocks
x8/

Deleted Files
a

x7

x5

tmp

usr

x4

bin

x1

x6

b

slg

ls

cp

mv

beth

mail

junk

x3

x2

Files

.

30

NO DATA: The disk is factory fresh.
.

Zero Blocks

Deleted Files
All Blocks are
Zero
Files

.

time

31

FORMATTED: The disk has an empty file system.

Zero Blocks

Deleted
Files
Blank
AllBlocks
Blocks are
Zero
Files
File System Structures
.

time

32

AFTER OS INSTALL: Temp. files have been deleted.

Zero Blocks

Deleted
Files
Blank
AllBlocks
Blocks are
Free Blocks
Zero
Files

Deleted temporary files

OS andApplications
File System
Structures
.

time

33

AFTER A YEAR OF SERVICE.

Blocks never written

Zero Blocks

Deleted files
Deleted
Files
Blank
AllBlocks
Blocksare...
... 1 year
Free Blocks
Zero
Files

OS, Applications,and user
Deleted temporary
files files

OS and Applications
File System Structures
.

time

34

DISK NEARLY FULL!
.

Blocks never written

Zero Blocks

Deleted files
OS, Apps,
DeletedFiles
Blankuser files,
AllBlocks
Blocksare...
... 1 year
Free Blocks
and lots of
Zero
MP3s!
OS, Applications,and user
Files
Deleted temporary
files files
OS andApplications
File System
Structures
.

time

35

FORMAT C:\ (to sell the computer.).

Blocks never written

Zero Blocks

Deleted files
OS, Apps,
DeletedFiles
Blankuser files,
AllBlocks
Blocksare...
... 1 year
Free Blocks
andlots of
Recoverable
Zero
MP3s!
Data
OS, Applications,and user
Files
Deleted temporary
files files
OS and Applications
File System Structures
.

time

36

We can use forensics to reconstruct motivations:.

OS, Apps,
Usability
user files,
failureandlots of
Recoverable
MP3s!
Data

Training
failure

.

time

37

Drives 1–236 are dominated by failed sanitization attempts.
2, 500

Megabytes

2, 0001, 500No Data (blocks cleared)
Data not in the file system (level 2 and 3)
Data in the file system (level 0)

1, 000500

0..but training failures are also important.
38

Overall numbers

Drives Acquired:
Drives DOA:
Drives Images:
Drives Zeroed:
Drives “Clean Formatted:”
Total files:
Total data:

236
60
176
11
22
168,459
125G

39

Only 33 out of 176 working drives were properly cleared!
• 1 from Driveguys — but 2 others had lots of data.
• 18 from pcjunkyard — but 7 others had data.
• 1 from a VA reseller — 1 DOA; 3 dirty formats.
• 1 from an unknown source — 1 DOA, 1 dirty format.
• 1 from Mr. M. who sold his 2GB drive on eBay.

40

MD5 hashing allows the identification of files.
Interestingly, few unique files that had not been deleted:
File type
Unique Files
Microsoft Word files:
783
Microsoft Excel files:
184
Microsoft PowerPoint files:
30
Outlook PST files:
11
audio files:
977

Conclusion: most users DELeted their files before
discarding their drives.

41

But what really happened?

?
I needed to contact the original drive owners.
42

The Remembrance of Data Passed Traceback Study.
[Garfinkel 05]
1. Find data on hard drive
2. Determine the owner
3. Get contact informationfor organization
4. Find the right person
inside the organization
5. Set up interviews

06/19/1999 /:dir216/Four H Resume.doc
03/31/1999 /:dir216/U.M. Markets & Society.doc
08/27/1999 /:dir270/Resume-Deb.doc
03/31/1999 /:dir270/Deb-Marymount Letter.doc
03/31/1999 /:dir270/Links App. Ltr..doc
08/27/1999 /:dir270/Resume=Marymount U..doc
03/31/1999 /:dir270/NCR App. Ltr..doc
03/31/1999 /:dir270/Admissions counselor, NCR.doc
08/27/1999 /:dir270/Resume, Deb.doc
03/31/1999 /:dir270/UMUC App. Ltr..doc
03/31/1999 /:dir270/Ed. Coordinator Ltr..doc
03/31/1999 /:dir270/American College ...doc
04/01/1999 /:dir270/Am. U. Admin. Dir..doc
04/05/1999 /:dir270/IR Unknown Lab.doc
04/06/1999 /:dir270/Admit Slip for Modernism.doc
04/07/1999 /:dir270/Your Honor.doc

6. Follow guidelines forhuman subjects work

This was a lot harder than I thought it would be.
43Ultimately, I contacted 20 organizations between April 2003
and April 2005.

44

The leading cause: betrayed trust.
Trust Failure: 5 cases
4 Home computer; woman’s son took to “PC Recycle”
4 Community college; no procedures in place
4 Church in South Dakota; administrator “kind of crazy”
4 Auto dealership; consultant sold drives he “upgraded”
4 Home computer, financial records; same consultant

This specific failure wasn’t considered in [GS 03];
it was the most common failure.

45Second leading cause: Poor training and supervision
Trust Failure: 5 cases
Lack of Training: 3 cases
4 California electronic manufacturer
4 Supermarket credit-card processing terminal
4 ATM machine from a Chicago bank

Alignment between the interface and the underlying
representation would overcome this problem.

46

Sometimes the data custodians just don’t care.
Trust Failure: 5 cases
Lack of Training: 3 cases
Lack of Concern: 2 cases
4 Bankrupt Internet software developer
4 Layoffs at a computer magazine

Regulation on resellers might have prevented these cases.

47

In seven cases, no cause could be determined.
Trust Failure: 5 cases
Lack of Training: 3 cases
Lack of Concern: 2 cases
Unknown Reason: 7 cases
8 Bankrupt biotech startup
8 Another major electronics manufacturer
8 Primary school principal’s office
8 Mail order pharmacy
8 Major telecommunications provider
8 Minnesota food company
8 State Corporation Commission

Regulation might have helped here, too.
48

“Deleted” data can be recovered in other areas
many of these sources, their credibility was difficult to assess and was often left to the foreign
government services to judge. Intelligence Community HUMINT efforts against a closed society
like Iraq prior to Operation Iraqi Freedom were hobbled by the Intelligence Community's
dependence on having an official U.S. presence in-country to mount clandestine HUMINT
collection efforts.
(U) When UN inspectors departed Iraq, the placement of HUMINT agents and thedevelopment of unilateral sources inside Iraq were not top priorities for the Intelligence
Community. The Intelligence Community did not have a single HUMINT source collecting
against Iraq's weapons of mass destruction programs in Iraq after 1998. The Intelligence
Community appears to have decided that the difficulty and risks inherent in developing sources
or inserting operations officers into Iraq outweighed the potential benefits. The Committee
found no evidence that a lack of resources significantly prevented the Intelligence Communityfrom developing sources or inserting operations officers into Iraq.

Document Files

When Committee staff asked why the CIA had not considered
placing a CIA officer in Iraq years before Operation Iraqi Freedom to investigate Iraq's weaponsof mass destruction programs, a CIA officer said, "because it's very hard to sustain . . . it takes arare officer who can go in . . . and survive scrutiny | ^ | [ m | | | for a long time." TheCommittee agrees that such operations are difficult and dangerous, but they should be within thenorm of the CIA's activities and capabilities. Senior CIA officials have repeatedly told theCommittee that a significant increase in funding and personnel will be required to enable to the
CIA to penetrate difficult HUMINT targets similar to prewar Iraq. The Committee believes,
however, that if an officer willing and able to take such an assignment really is "rare" at the CIA,
the problem is less a question of resources than a need for dramatic changes in a risk averse
corporate culture.
(U) Problems with the Intelligence Community's HUMINT efforts were also evident in
the Intelligence Community's handling of Iraq's alleged efforts to acquire uranium from Niger.
The Committee does not fault the CIA for exploiting the access enjoyed by the spouse of a CIA
employee traveling to Niger. The Committee believes, however, that it is unfortunate,considering the significant resources available to the CIA, that this was the only option available.
Given the nature of rapidly evolving global threats such as terrorism and the proliferation of
weapons and weapons technology, the Intelligence Community must develop means to quickly
respond to fleeting collection opportunities outside the Community's established operating areas.
The Committee also found other problems with the Intelligence Community's follow-up on the-25-

Web Browsers

49

Information is left in document files.

• The New York Times published a PDF filecontaining the names of Iranians whohelped with the 1953 coup. [Young 00]• US DoJ published a PDF file “diversity
report” containing embarrassing redacted
information. [Poulsen 03]• SCO gave a Microsoft Word file tojournalists that revealed its Linux legal
strategy. [Shankland 04]• Multinational Force-Iraq report50

The information leaked because two patterns were not
implemented.
Visibility

Sanitization





Users

Users

User
Audit

Explicit Item
DeleteReset toInstallation
Complete
DeleteDelayedUnrecoverable
ActionDocument Files, Applications, and Media

51

The Senate Foreign Intelligence Committee prevented
leakage by scanning its redacted report on pre-war Iraq
intelligence failures to create the PDF that it distributed.

This violates Section 503 (but they don’t care).
52

Microsoft has tried to solve this problem with its“Remove Hidden Data” tool.

RHD doesn’t integrate into the flow of document
preparation. The patterns-based analysis predicts that RHD
will fail in many cases.
53

Information is left behind in web browsers.

Browser
History

Cookies

BrowserCache

Two key problems: À Deleted files; Á The cache54

In fact, a lot of information is left behind in web browsers.

MIT Humanities Library, April 25, 2005

55

4 out of 4 computers had personal email in their browser
caches.

The American Library Association recommends softwarethat automatically purges caches on a daily basis.
(It would be better to purge after each use.)

56

Legislative reactions to this research:“Fair and Accurate Credit Transactions Act of 2003” (US)• Introduced in July 2003.
Signed December 2003.
• Regulations adopted in 2004, effective June 2005.
• Amends the FCRA to standardize consumer reports.
• Requires destruction of paper or electronic “consumer
records.”

Testimony: http://tinyurl.com/cd2my57

Technical reactions to this research:“Secure Empty Trash” in MacOS 10.3.

58

Unfortunately, “Secure Empty Trash” is incomplete.

• Implemented in Finder(inconsistently)• Locks trash can• Can’t change your mind

59

MacOS 10.4 “Erase Free Space” makes a big file.

60

MacOS “File Vault” gives users an encrypted file system.

61

Future Work: Deploying Compete Delete• Make FORMAT actually erase the disk.
• Make “Empty Trash” actually overwrite data.
• Integrate this functionality with webbrowsers, word processors, operating
systems.
• Address usability dangers of clean delete.
• Analysis of “one big file” technique.

many of these sources, their credibility was difficult to assess and was often left to the foreign
government services to judge. Intelligence Community HUMINT efforts against a closed society
like Iraq prior to Operation Iraqi Freedom were hobbled by the Intelligence Community's
dependence on having an official U.S. presence in-country to mount clandestine HUMINT
collection efforts.
(U) When UN inspectors departed Iraq, the placement of HUMINT agents and thedevelopment of unilateral sources inside Iraq were not top priorities for the Intelligence
Community. The Intelligence Community did not have a single HUMINT source collecting
against Iraq's weapons of mass destruction programs in Iraq after 1998. The Intelligence
Community appears to have decided that the difficulty and risks inherent in developing sources
or inserting operations officers into Iraq outweighed the potential benefits. The Committee
found no evidence that a lack of resources significantly prevented the Intelligence Communityfrom developing sources or inserting operations officers into Iraq.

When Committee staff asked why the CIA had not considered
placing a CIA officer in Iraq years before Operation Iraqi Freedom to investigate Iraq's weaponsof mass destruction programs, a CIA officer said, "because it's very hard to sustain . . . it takes arare officer who can go in . . . and survive scrutiny | ^ | [ m | | | for a long time." TheCommittee agrees that such operations are difficult and dangerous, but they should be within thenorm of the CIA's activities and capabilities. Senior CIA officials have repeatedly told theCommittee that a significant increase in funding and personnel will be required to enable to the
CIA to penetrate difficult HUMINT targets similar to prewar Iraq. The Committee believes,
however, that if an officer willing and able to take such an assignment really is "rare" at the CIA,
the problem is less a question of resources than a need for dramatic changes in a risk averse
corporate culture.
(U) Problems with the Intelligence Community's HUMINT efforts were also evident in
the Intelligence Community's handling of Iraq's alleged efforts to acquire uranium from Niger.
The Committee does not fault the CIA for exploiting the access enjoyed by the spouse of a CIA
employee traveling to Niger. The Committee believes, however, that it is unfortunate,considering the significant resources available to the CIA, that this was the only option available.
Given the nature of rapidly evolving global threats such as terrorism and the proliferation of
weapons and weapons technology, the Intelligence Community must develop means to quickly
respond to fleeting collection opportunities outside the Community's established operating areas.
The Committee also found other problems with the Intelligence Community's follow-up on the-25-

Questions?
62