Report to the Chairman, Committee on

Government Reform, House of

Representatives

United States General Accounting Office

GAO

April 2001 ELECTRONIC

RECORDS

Clinton

Administration’s

Management of

Executive Office of

the President’s E-Mail

System

GAO-01-446
Page i GAO-01-446 EOP E-mail System

Letter 1

Appendix I Scope and Methodology 19

Appendix II Chronology of E-mail Malfunctions 21

Appendix III Key Officials Involved in Primary Mail2 and

Letter D Events 39

Appendix IV Time Line of Key Events in the Mail2 and Letter D

Malfunctions 41

Appendix V Itemization of Budgeted Cost for the Tape

Restoration Project 42

Appendix VI Comments From the Office of the Former Vice

President 43

Tables

Table 1: Summary of Budgeted Tape Restoration Costs. 9

Table 2: Additional Project Activities and Related Costs. 15

Contents
Page 1 GAO-01-446 EOP E-mail System

April 30, 2001

The Honorable Dan Burton

Chairman, Committee on Government Reform

House of Representatives

Dear Mr. Chairman:

In response to your requests of May and August 2000, this report discusses

our examination of the facts surrounding key events affecting the ability of

the Clinton Administration’s Executive Office of the President (EOP) to

preserve certain e-mail messages deemed official government records. As

you know, the Presidential Records Act (PRA) and the Federal Records

Act (FRA) require the EOP to preserve official records, including e-mail.

Adequate controls must be in place to ensure that these records, whether

manual or electronic, are complete and maintained in accordance with

applicable requirements. However, e-mail system malfunctions and

management weaknesses prevented archiving of some e-mail records for

EOP components, including the Office of the Vice President (OVP).

In our review of these events surrounding the EOP’s e-mail systems, you

asked us to:

• develop a chronology of the e-mail malfunctions that included a

description of EOP actions taken in discovering and repairing the

malfunctions and recovering missing e-mail messages;

• identify officials and contractors responsible for maintaining the e-mail

system and correcting the malfunctions;

• determine whether the OVP implemented adequate practices in

accordance with applicable laws, regulations, and generally accepted

industry practices for management of e-mail records;

• identify elements contributing to the estimated cost to repair the system

and restore lost messages; and

• determine whether EOP officials notified government officials—including

the Congress, the Justice Department, and the Office of Independent

Counsel—when the malfunctions were discovered.

To address these objectives, we reviewed documents submitted to your

Committee by the EOP. In addition, we observed July through October

2000 U.S. District Court hearings pertaining to EOP e-mail. We also

submitted written questions to the EOP, and reviewed the written

United States General Accounting Office

Washington, DC 20548
Page 2 GAO-01-446 EOP E-mail System

responses and additional documentation provided to us. We performed

our work from July 2000 through January 2001, in accordance with

generally accepted government auditing standards; however, our review

was limited by the unavailability of complete information pertaining to the

OVP’s retention of hard copy e-mail records, the EOP’s assessment of

contractor performance, and the EOP’s notification of government

officials. A detailed discussion of the scope and methodology for

conducting our work is presented in appendix I.

Two malfunctions that occurred in the EOP e-mail system prevented

official records from being properly recorded in the Automated Records

Management System (ARMS), which is a searchable database of e-mail

records that was implemented by the EOP in 1994. The first—an anomaly

with incoming Internet e-mail to affected users of the “Mail2” e-mail

server—was identified in late January 1998. In November 1998, in

addressing the Mail2 repair, an EOP contractor introduced a second

problem that prevented incoming e-mail to users with first names starting

with the letter D from being captured by ARMS. Although these

malfunctions prevented certain e-mail records from being archived in

ARMS, copies of these records were retained in the system backup tapes

that were saved between November 1998 and June 1999 to ensure that the

system could be restored after an interruption in operation. To ensure that

official government records were preserved in a searchable format, the

EOP initiated a tape restoration project in March 2000 to retrieve e-mail

records from available backup tapes and add these records to ARMS. A

detailed chronology of these events is presented in appendix II. Appendix

III contains a list of federal officials and contractors involved in key events

surrounding maintenance of the e-mail systems and repair of the Mail2 and

Letter D malfunctions.

The OVP did not implement adequate records management practices to

ensure that all e-mail records generated or received were preserved in

accordance with applicable law and best practices. The OVP initially

followed a dual approach to managing its e-mail records pursuant to the

EOP policy—maintaining paper copies of records as well as retaining

backup e-mail tapes. The OVP discontinued this approach at some point

after May 1993—in the mistaken belief that the e-mail records were being

archived in ARMS and that EOP’s Office of Administration was managing

the backup tapes. The OVP could not demonstrate that all e-mail records

had been preserved by acceptable methods until May 2000, when ARMS

began capturing all OVP e-mail records. As a result, about 600 system

Results in Brief
Page 3 GAO-01-446 EOP E-mail System

backup tapes needed restoration to determine if any non-archived e-mail

records existed on the tapes.

Several factors contributed to the expected cost to restore omitted EOP

e-mail records to ARMS, estimated by an EOP contractor to be $11.7

million:

• An EOP support contractor’s performance of tape management and

systems maintenance and documentation activities contributed to the

increased size and scope of the tape restoration project. Specifically, the

EOP’s assessments of contractor performance revealed that management

of e-mail backup tapes was weak and that maintenance of e-mail systems

and user accounts needed improvement. In addition, the automated link

between the EOP’s e-mail system and ARMS was not documented as

required. As a result, the EOP could not identify the specific tapes within

its entire tape population that needed restoration and could not readily

understand and optimize records management controls.

• The EOP did not effectively monitor management of e-mail records. It did

not include critical elements in its monitoring program, such as

evaluations of the creation of records, maintenance and use of records,

and records disposition. This increased the time needed to detect the

Mail2 and Letter D malfunctions and added to the accumulation of backup

tapes to be restored.

• Scrutiny of the e-mail malfunctions and the EOP’s tape restoration

practices by external authorities introduced additional project tasks,

including contracts for independent verification and validation of the

restoration process and for additional security over the project.

Although the EOP knew of the malfunctions that prevented e-mail from

being captured by ARMS from October 1996 through mid-May 1999,

officials stated that they did not understand the breadth and scope of the

impact of these malfunctions on the EOP’s response to subpoenas and

production of other documents until February 2000. The EOP claims that

it notified appropriate investigative bodies at that time. However, we were

unable to obtain sufficient evidence to confirm that these notifications

were made.

We requested comments on a draft of this report from the EOP and the

personal representatives of former President Clinton and former Vice

President Gore. The Deputy General Counsel, Office of Administration,

EOP, orally provided a technical clarification, which was incorporated.

The Counsel to former Vice President Gore provided written comments

stating that (1) the report did not explicitly state that we found no
Page 4 GAO-01-446 EOP E-mail System

evidence of attempts by OVP staff to deliberately fail to preserve records,

and (2) ARMS was a unique system for which best practices did not apply

and additional information on the creation and use of this system was

provided. In regard to the statements regarding the intent of OVP staff, our

review—as requested—focused on the OVP’s management of e-mail

records. We did not assess the intent of individuals responsible for

managing these records and, as such, made no conclusions on this subject.

Regarding ARMS, we disagree with the Office of the former Vice

President’s characterization of the applicability of best practices to ARMS.

The best practices cited in this report currently exist in the form of

National Archives and Records Administration guidance and apply to

records management programs, irrespective of the type of system used.

However, we have revised the report to include the additional information

on ARMS that was provided. The Office of the former Vice President also

provided technical comments that we have incorporated as appropriate.

The representative of former President Clinton provided a letter, which

stated that he did not have access to the records necessary to properly

review and comment on its accuracy.

PRA, set forth in title 44 of United States Code, chapter 22, requires the

President and Vice President to adequately record their official acts,

maintain certain official records, and transfer custody of such records to

the Archivist of the United States upon termination of their terms of office.

Pursuant to the PRA, both the Office of the President and the Office of the

Vice President are to implement records management controls and other

necessary actions to ensure that presidential and vice presidential

activities, deliberations, decisions, and policies are adequately

documented and maintained. It also provides that presidential records

shall be made available pursuant to subpoena or other judicial process and

to either House of Congress, and prohibits destruction of records without

prior concurrence of the Archivist and notification of Congress 60 days

prior to disposal. The Archivist promulgates standards and guidance for

implementation of PRA; these are contained in title 36 of the Code of

Federal Regulations (CFR). This regulation defines criteria for access to

and disposition of presidential records.

Federal records are subject to the provisions of FRA, set forth in title 44

U.S.C., chapters 29, 31, and 33. FRA requires the heads of federal agencies

to make and preserve records documenting the official activities of the

agency. FRA directs federal agencies to establish (1) a program for the

management of agency records, (2) effective controls over the creation,

maintenance, and use of records, and (3) safeguards against the removal

Background
Page 5 GAO-01-446 EOP E-mail System

or loss of records. The Archivist promulgates standards and guidance for

implementation of FRA, which are contained in title 36 CFR. Requirements

for federal records are more extensive than are those for presidential

records, and specify that federal agencies

• preserve records documenting the official activities of federal agencies to

protect the legal rights of the government and of persons directly affected

by the agency’s activities;

• periodically monitor staff determinations of the record status of

documentary materials;

• implement a records maintenance program so that complete records are

filed or otherwise identified and preserved and that records can be found

when needed;

• implement an agencywide program for the management of all federal

records created, received, and stored on electronic media;

• ensure compliance with federal criteria and review electronic information

systems periodically for conformance with established agency procedures,

standards, and policies;

• develop and maintain up-to-date documentation about all electronic

information systems adequate to specify technical characteristics,

understand the purpose and function of the systems, and ensure timely,

authorized disposition of records;

• consider specific criteria when developing procedures for maintenance of

electronic mail records in recordkeeping systems, including copying

records from the e-mail system to a separate recordkeeping system;1

 and

• retain records from e-mail systems in an off-line storage format. Agencies

that use electronic formats (such as optical disk or magnetic tape) must

maintain the ability to convert the records to the National Archives and

Records Administration’s required format and medium at the time of

transfer. Agencies that maintain paper files as their recordkeeping systems

are required to print their electronic mail records and the related

transmission and receipt data.

To ensure the adequacy of management and control of federal information

systems, the Office of Management and Budget (OMB) issued circulars A123, Management Accountability and Control, and A-130, Management of

Federal Information Resources. Circular A-123 requires that federal

 1

A recordkeeping system is a manual or automated system in which records are collected,

organized, and categorized to facilitate their preservation, retrieval, use, and disposition.

Because they do not have the features specified by the National Archives and Records

Administration, system backup tapes are not to be used for recordkeeping purposes.
Page 6 GAO-01-446 EOP E-mail System

agencies implement adequate internal controls and management

structures to ensure the effective operation of federal programs. Appendix

III of circular A-130,2 Security of Federal Automated Information

Resources, requires that federal agencies periodically test system controls

meant to ensure the integrity, confidentiality, and availability of

information resources.

The Administration issued guidance to clarify the responsibilities of each

EOP component regarding presidential, federal, and personal records in

order to implement the provisions of both PRA and FRA. A detailed policy

and procedures document issued by the EOP in May 1993 required that no

federal or presidential e-mail records could be deleted unless maintained

in an electronic recordkeeping system or printed and placed in a file. In

addition, it required that the White House Records Management Office

monitor electronic systems to ensure that correct record status

determinations had been made for presidential and federal e-mail records.

This policy was applicable to all EOP components,3

 including the OVP.

In May 1997, the White House replaced the PRA section of the 1993 policy

with one that provided a summary of White House policies and guidance

regarding PRA. It required preservation of all original presidential records

and all materials covered by subpoena or other such requests. It also

directed each staff member to determine the record status of outgoing email messages and to maintain hard copy records in organized files.

According to EOP officials, the policy issued in 1997 was substantively the

same as the 1993 policy, and it served as a reminder to staff of the policy

originally articulated in the 1993 policy regarding compliance with PRA.

A series of events occurred between July 1994, when ARMS was

implemented, and March 2000, when the tape restoration project was

initiated. These events are detailed in appendix II. Various EOP federal

and contractor staff were involved in the maintenance and repair of the

 2

OMB circular A-130 applies to EOP’s Office of Administration component, which had

responsibility for electronic systems, including e-mail and ARMS.

3

The EOP consists of the White House Office, the OVP, and other components, such as

OMB and the Council on Environmental Quality. EOP components that create and receive

presidential records included the White House Office, the OVP, the National Security

Council, and the Office of Policy Development.

The EOP Policies

Required Management

of Records

History Surrounding

the EOP E-mail

Systems
Page 7 GAO-01-446 EOP E-mail System

e-mail systems between October 1996 and June 1999; they are listed in

appendix III.

According to the EOP, in order to comply with the Armstrong ruling,4

 it

contracted with Information Management Consultants, Incorporated, to

develop the ARMS recordkeeping system. ARMS began archiving e-mail

records in 1994.5

Since 1996, the EOP had used Lotus Notes® software as its e-mail

application. During the former administration, the Office of Administration

within the EOP maintained four Lotus Notes e-mail servers, one remote

server, and one ARMS interface server, which transferred e-mail records

from the e-mail system to ARMS.

E-mail messages designated as records subject to either PRA or FRA were

passed to ARMS through the interface between the two systems. The

Notes-to-ARMS interface program automatically identified unrecorded

e-mail messages when it periodically scanned for new messages in

groupings of user accounts called “views.” User accounts were distributed

to a view depending on the first letter of the account name. Then, the

ARMS interface copied these new messages for processing to ARMS and

marked the messages as “recorded.” In its comments on a draft of this

report, the Office of the former Vice President stated that the ARMS

database maintained e-mail records in a searchable format, which came to

facilitate the EOP’s responses to document requests and subpoenas

sometime much later in the administration. A key feature included in

ARMS was the ability to perform a computerized search of e-mail using

keyword terms to search for responsive materials.

From October 1996 through May 1999, two malfunctions occurred in the

EOP e-mail system that prevented incoming Internet e-mail from being

properly archived in ARMS. In late January 1998, an EOP employee

initially identified the first of these malfunctions—an anomaly with

incoming Internet e-mail to affected users of the Mail2 e-mail system

server—and documented this in an incident report. According to EOP

 4

Armstrong v. EOP, 810 F. Supp. 335 (1993). The court in the Armstrong case held that the thencurrent EOP records management practice did not save all information from an electronic record, and

thus violated FRA.

5

The EOP initially used ARMS with the e-mail system that preceded use of the Lotus

NotesÆ e-mail system.
Page 8 GAO-01-446 EOP E-mail System

officials, efforts were undertaken then to analyze this anomaly, but it was

thought to be an isolated event. According to these officials, the precise

cause of the Mail2 malfunction was determined in June 1998 by Northrop

Grumman, the EOP’s information technology services support contractor

since October 1997, when performing other work on the Lotus Notes email system interface to the ARMS records management database. The

malfunction was caused by improper user account configuration in which

the server name “Mail2” was spelled “MAIL2.” Because the ARMS interface

program did not recognize the upper case spelling of the mail server name,

it was unable to locate and capture new incoming e-mail messages for

these user accounts. This malfunction was subsequently repaired in

November 1998.

Also in November 1998, in addressing the Mail2 repair, a second problem

was introduced by Northrop Grumman staff that prevented incoming

Internet e-mail to users with first names starting with the letter D from

being captured by ARMS. It involved omission of the letter D from the

Lotus Notes view (described above), causing the ARMS scanner to skip

those accounts beginning with the letter D and not capture incoming email to the affected user accounts. This malfunction was discovered in

April 1999 and repaired in May 1999. Appendix IV presents a time line of

key events pertaining to the discovery and repair of the Mail2 and Letter D

malfunctions.

The EOP e-mail systems were periodically backed up, or copied, to tape

media for use in the event of system failure and subsequent need to

restore the system to full operation. In addition, Northrop Grumman

followed a standard industry practice of economical tape recycling. This

involved retaining the tapes for 3 weeks and then recycling them—

resulting in either overwriting the data on the tapes or destroying the used

tape and replacing it with a new tape. Because the backup tapes contained

copies of e-mail not processed to the ARMS records management

database, the EOP stopped its normal practice of tape recycling from June

1998, when the cause of the Mail2 malfunction was documented, until June

1999, after the Letter D malfunction was repaired.

In March 2000, the EOP implemented the tape restoration project, for

which it hired Enterprise Computing Solutions Technology, Incorporated

(ECS) to recover the lost e-mail messages and restore the messages to the

ARMS records management database. The EOP hired another contractor,

Vistronix, to oversee the work of ECS. In July 2000, Vistronix estimated

that the project would cost about $11.7 million for restoration of about
Page 9 GAO-01-446 EOP E-mail System

6,000 tapes. Table 1 presents a breakdown of the $11.7 million budgeted

cost. Appendix V presents further details on this.

Table 1: Summary of Budgeted Tape Restoration Costs.

Budgeted cost

element Description of cost element

Estimated

cost

ECS contract Restoration of e-mail messages not recorded in

ARMS $5,125,829

Vistronix contract Independent validation and verification of the

tape restoration project 1,993,819

AAC Associates

contract

Advice on recovery of OVP e-mail

900,000

Special services Forensic analysis and recovery of broken or

unreadable tapes 2,092,400

Direct costs Additional hardware, software licenses, and

12,000 tapes for two sets of tape copies 498,480

Other Project overrun contingency 1,098,891

Total $11,709,419

Source: Tape restoration project cost estimate report prepared by Vistronix, Incorporated, for the

EOP in July 2000.

According to the EOP, although the 1993 EOP records management policy

applied to the OVP, the OVP did not implement adequate practices to

ensure continual preservation of its e-mail records in accordance with

PRA, which required implementation of records management controls.

The OVP’s internal policy from 1993 until some time in 1998 was twofold—

to rely on its staff to retain its e-mail server backup tapes and to maintain

files of hard copy e-mail records.6

 According to the EOP, however, from

May 1993 until May 2000, the OVP could not demonstrate that its practices

for managing presidential e-mail records provided continual preservation

of records during the former administration.

During a portion of the administration, the OVP was responsible for

operation, maintenance, and records management of its own e-mail system

separate from that used by the EOP. In March 1998, responsibility for

operation and maintenance of the system transferred to the Office of

 6

According to EOP, a December 21, 2000, letter from EOP to the National Archives and

Records Administration documented agreement reached regarding OVP’s records

management approach. However, our review of this letter, obtained from the National

Archives and Records Administration, found the letter to discuss disposition of e-mail

records upon the presidential transition and it did not mention the OVP records

management approach.

The OVP Did Not

Ensure Adequate

Preservation of E-mail

Records
Page 10 GAO-01-446 EOP E-mail System

Administration. However, the OVP did not communicate its requirements

for management of e-mail records and assumed that records management

responsibility also transitioned to the Office of Administration. According

to EOP, the Office of Administration assumed that the OVP had retained

records management responsibilities and did not take steps to ensure that

OVP e-mail records were appropriately preserved.

According to the EOP, “at some subsequent point” since May 1993, the

OVP discontinued retention of paper copies of e-mail records because it

believed its e-mail to be captured by ARMS. This mistaken belief was

based on the precedent that ARMS managed e-mail records for all other

EOP components since its implementation in 1994. In its comments on a

draft of this report, the Office of the former Vice President stated that this

belief was also based on the fact that searches of ARMS produced OVP email records and on the observation that some outgoing OVP e-mails were

in fact being managed by ARMS (as discussed below). The EOP estimated

that “by 1998,” the OVP discontinued retention of paper copies of e-mail

records not captured by ARMS (but subject to PRA). Therefore, paper

copies of e-mail records did not exist for a period of at least 2 years (1998

and 1999) and up to 7 years (1993 through 1999). In its comments on a

draft of this report, the Office of the former Vice President stated that

these records are being restored as part of the tape restoration project. We

requested access to files of any hard copy e-mail records covering this

period, but it was not provided.

Although the OVP did not continually retain paper copies of e-mail records

in accordance with the EOP policy, it did retain the backup tapes created

through late March 1998 (except for a backup malfunction from midFebruary to mid-March 1994). According to the EOP, the OVP backup

tapes were for system recovery and records archival, but they were not in

a text-searchable format.7

After transition of the e-mail system in March 1998, the Office of

Administration maintained the OVP e-mail system in accordance with its

established operating practices for other EOP systems. One practice

followed was to recycle system backup tapes every 3 weeks. Application

of this practice to the OVP system resulted in irretrievable loss of the

 7

According to the EOP in June 2000, the OVP tapes were saved for compliance with PRA. In

January 2001, the EOP stated that the OVP did not routinely search its backup tapes in

response to subpoenas and that only through the tape restoration project has the OVP

demonstrated that the tapes are recoverable.
Page 11 GAO-01-446 EOP E-mail System

electronic form of e-mail records while tapes were recycled between

March 1998 and March 2000. Another practice was to use the standard

EOP ARMS e-mail template8

 to create new OVP e-mail user accounts

between March 1998 and March 2000, which inadvertently caused ARMS

to capture outgoing e-mail records sent by these users. The result of this

practice was that ARMS captured outgoing e-mail for those OVP users that

had the EOP e-mail template (133 of 157 OVP users). When it learned in

March 2000 that ARMS did not fully preserve OVP e-mail records, the EOP

ordered that tape recycling be stopped and initiated corrective action.

According to the EOP, all OVP e-mail users, except Senate OVP staff,9

became fully recorded in ARMS as of May 8, 2000. However, because the

OVP did not ensure the preservation of e-mail records, in either paper or

electronic form, presidential records may have been irretrievably lost. In

addition, about 600 backup tapes were added to the restoration project to

recover as many e-mail records as possible.

Several factors affected the size and scope of the tape restoration project

and the related estimated cost of $11.7 million to restore omitted e-mail

records to ARMS. First, an EOP support contractor’s performance led to

increases in the number of tapes for restoration and adding to the passage

of time necessary to diagnose system anomalies and repair malfunctions.

Second, the EOP’s monitoring of e-mail records did not provide reasonable

assurance that ARMS captured all required records, which increased the

time needed to detect the Mail2 and Letter D malfunctions and added to

the accumulation of backup tapes for the restoration effort. Finally, tasks

were added to the project due to legal scrutiny of the e-mail malfunctions

and the EOP’s tape restoration practices.

In supporting its information technology functions, the EOP had a primary

support services contract, which required the contractor to provide

common services to all users of the EOP data center. The data center

provides general computer operations, server and network support, and

software support. Under its contract, Northrop Grumman was required to

 8

The ARMS e-mail template was configured to automatically pass to ARMS a blind copy of

all e-mail records sent by an EOP e-mail user.

9

Because the Vice President also serves as the President of the Senate, some OVP staff

support the Vice President’s Senate duties. Some Senate OVP staff used a separate e-mail

system controlled by the Senate.

Several Factors Have

Driven Cost of E-mail

Restoration

The EOP’s Support

Contractor Did Not

Effectively Perform Tasks

Related to E-mail Systems
Page 12 GAO-01-446 EOP E-mail System

• “manage cartridge tapes, monitor catalog assignments, maintain

appropriate tape management systems (automated on IBMs and VAXs,

manual on all other platforms),10 [and] maintain back up system tapes in

accordance with Government standards/procedures” as part of its

management of the data center tape library;

• develop and support Notes e-mail, administer Notes systems including

server hardware and software configuration, maintain mail backup

systems, resolve hardware and software problems for the ARMS interface,

provide technical support for operation and maintenance of the interface,

and develop and implement quality assurance programs for this work; and

• develop system documentation to capture the functions, interface, and

internal control requirements, and document requirements of existing

systems that need updating or were not previously documented.

The contract also included provisions for EOP oversight and monitoring of

contractor performance. The EOP established contract performance

measures and an associated performance evaluation plan, which provided

a mechanism for formal measurement and documentation of contractor

performance and allocating award fees and penalties based on

performance. The EOP conducted formal award fee assessments of

contractor performance twice per year, in accordance with the contract.

In monitoring the contractor, the EOP found weaknesses in the

contractor’s performance of data center tape management tasks. The

EOP’s award fee assessment report from October 1998 to mid-April 1999

noted that tape management, in general, had been a weak point in the

contractor’s performance. Specifically, the report stated that the

contractor had not created an inventory of the e-mail system backup tapes,

and that these tapes continued to mount in an uncontrolled manner. As a

result, the EOP rated the contractor’s performance as marginal and

withheld about $10,400 in award fees for this and other data center tasks.

The EOP’s subsequent award fee assessment report for the period of midApril to September 1999 stated that there continued to be a lack of

effective tape management in the data center and rated the contractor’s

performance as unsatisfactory. We were not able to determine what, if

 10According to FRA, manual external labels or automated tape management systems for

tapes storing electronic records must provide unique identification, including the name of

the unit responsible for the data and system title. Other identifying information such as file

title(s), dates of creation, volume serial number, and software dependency must be

maintained but not necessarily attached to the tape.
Page 13 GAO-01-446 EOP E-mail System

any, award fee was withheld because the EOP did not provide us with

complete award fee documentation. The problems with tape management

were not subsequently resolved. In March 2000, the EOP tape restoration

project manager stated in a memorandum that e-mail backup tapes

needing restoration could not be identified and that all tapes that had been

retained would need to be examined to determine their contents.

The EOP also documented deficiencies in the contractor’s performance of

e-mail system and user account maintenance. According to the EOP, the

contractor had not implemented satisfactory quality controls to prevent or

detect mistakes in user account and system configurations. As a result, the

EOP did not detect the Letter D malfunction for 5 months and the Mail2

malfunction for almost 2 years. For example:

• The EOP’s award fee assessment report for October 1998 through midApril 1999 stated that the configuration error that caused the Letter D

malfunction “should not have occurred and [it] could have been fixed on

the spot had [the contractor implemented] proper testing procedures and

quality assurance measures been in place.” As a result, the EOP rated the

contractor’s performance in this area as unsatisfactory and withheld

$20,832 in award fees for this and other system life cycle tasks.

• The error in configuration of user accounts that caused the Mail2

malfunction—entering “MAIL2” instead of “Mail2”—continued even after it

was discovered in June 1998. In April 1999, the EOP found that this

configuration error had not only occurred continually on the Mail2 server

since October 1996, but also on another EOP e-mail server. The EOP

initiated a review of controls to detect such configuration errors in May

1999.

Finally, up-to-date system documentation for the e-mail system interface

to ARMS was not maintained, as required by the contract and under FRA.

Accurate system documentation and availability of source code facilitate

an understanding of how a system functions, which enables the system

owner and system administrator to diagnose system anomalies—such as

those experienced with the Mail2 and Letter D malfunctions—and

optimize system controls. However, although a subcontractor to a former

EOP contractor documented the system when it developed the interface

between the Lotus Notes and ARMS systems, the Office of Administration

staff found that both the system documentation and the programming

source code were missing in 1997. Until the source code was located in

1998 and the EOP contractor documented the system functions in 1999,

the EOP was unable to implement changes to the code or develop

automated programs to monitor the effective functioning of the interface.
Page 14 GAO-01-446 EOP E-mail System

These shortcomings in tape management, system and account

maintenance, and system documentation contributed to the time

necessary for the EOP to detect the malfunctions and ultimately hindered

the EOP’s identification of the e-mail backup tapes requiring restoration,

increasing the quantity of tapes that would have to be examined in the

project. From the time between identification and repair of the

malfunctions—for Mail2, June to November 1998; for Letter D, April to

May 1999—and restoration of the tapes that began in March 2000, the EOP

continued to accumulate backup tapes. This resulted in raising the total

number of tapes to be restored from 788 in December 1998 to about 4,500

in January 2001 (comprised of 3,900 Mail2 and Letter D tapes and 600 OVP

tapes).

While the EOP’s 1993 policy required that the White House Records

Management Office monitor electronic systems containing presidential

and federal e-mail records, the EOP’s records management program did

not provide reasonable assurance that adequate controls were in place and

functioning to monitor the preservation of such records. Such a

monitoring program should include evaluations of the creation of records,

maintenance and use of records, and records disposition to determine

compliance with established agency and federal recordkeeping

requirements.

Pursuant to its policy, the EOP implemented a monitoring program in July

1994 to identify systemic problems with records determinations made by

EOP e-mail users. The EOP’s monitoring program applied, however, only

to federal records and did not cover monitoring of presidential records as

required by the EOP policy. In addition, the monitoring program did not

include tests to ensure the completeness of federal and presidential e-mail

records captured by ARMS that are created by incoming e-mail messages

to the EOP e-mail users. Controls were specifically needed over these

incoming messages because the e-mail system allowed the EOP users to

delete incoming messages before they were automatically copied to

ARMS. This ability, coupled with the EOP’s normal practice of recycling

backup tapes, may have resulted in the irretrievable loss of the electronic

form of e-mail records. However, the EOP did not document or regularly

test the adequacy of records management controls until May 1999 after

repair of the Letter D malfunction, when the EOP implemented an

automated e-mail monitoring program. Because controls were not in place

until May 1999, the Letter D malfunction went undetected for 5 months,

adding additional backup tapes to the size of the restoration project.

Monitoring Program

Did Not Ensure

Effective Management

of E-mail Records
Page 15 GAO-01-446 EOP E-mail System

Scrutiny over the tape restoration project by the U.S. District Court,11

Office of the Independent Counsel, and the Congress, began in March

2000, introducing additional tasks. Examples of such tasks include use of

law enforcement imaging software (required by the Office of the

Independent Counsel) and development of an interim searchable database

and hiring of an independent validation and verification contractor and

security guard support to enhance integrity of tape processing (required

by the U.S. District Court and the Congress).

According to the EOP, the restoration project was initiated in March

2000—not in 1998 after repair of Mail2 or in 1999 after repair of Letter

D—because the EOP’s focus in 1999 was on remediation of its systems to

ensure continued processing after the year 2000 and leap day (February

29, 2000) rollovers. As a result, the quantity of e-mail messages requiring

restoration escalated, drawing the attention of authorities interested in

ascertaining whether the EOP had implemented adequate controls. Table 2

lists restoration project activities estimated at about $3.5 million that were

introduced because of this scrutiny.

Table 2: Additional Project Activities and Related Costs.

Project activity Cost (estimated)

Independent verification and validation contractor $1,993,819

Security (two contracts):

 General Services Administration

 TW & Company

(actual) 97,920

54,894

Development of interim searchable database 963,400

Tape inventory and catalogue (list) tape contents 455,000

Total $3,565,033

Source: Tape restoration project cost estimate report prepared by Vistronix, Incorporated, for the

EOP in July 2000, the EOP summary of actual costs through October 29, 2000, and discussions with

EOP officials.

 11The tape restoration project was being monitored by Judge Lamberth of the U.S. District Court

incident to the case of Alexander v. FBI (CA 96-2125, D.D.C.). The plaintiffs in that case had

submitted document requests to the EOP in an attempt to discover relevant information in EOP’s email.

Legal Scrutiny Added

Restoration Tasks
Page 16 GAO-01-446 EOP E-mail System

The EOP claimed to have notified investigative bodies as soon as it

understood that document production might have been affected by the email malfunctions. Although the cause of the Mail2 e-mail malfunction was

known as early as June1998 and Letter D in April 1999, the White House

Counsel’s Office stated that it did not discover until February 2000 that the

e-mail malfunctions “had affected the integrity of White House document

productions.” Thus, “the Counsel’s Office did not perceive a need to notify

investigative bodies” prior to that time.

According to EOP officials, when the extent of the malfunctions became

clear to the White House Counsel’s Office in February 2000, that office

“had both written and oral communications with various investigative

bodies regarding the Mail2 issue.” The EOP provided us with copies of its

correspondence to the House Committee on Government Reform that

range in time from March 17 to October 30, 2000. On March 17, 2000, the

EOP provided the Committee with a description of the e-mail system, the

malfunctions, and their effect on document production. A letter to the

Committee in June 2000 updated the status of e-mail records management

for the Office of the Vice President. Finally, an October 6, 2000, letter from

the EOP to the Committee referred to 30 letters from the EOP to various

investigative bodies, between March 15 and October 3, 2000, including the

Offices of Independent Counsel, congressional committees, and the

Department of Justice.

We requested copies of these 30 letters, but we were not given copies of

any of them. Without complete documentation, we were unable to confirm

the EOP’s claims to have notified officials concerning the effect of the email malfunctions on respective document productions to investigative

bodies other than the Committee. In its comments on a draft of this report,

the Office of the former Vice President stated that as a matter of policy,

“the EOP did not cross-pollinate its communications with various

investigative bodies as a means of maintaining the confidentiality of the

legitimate interests and investigative priorities of those investigative

bodies.”

Computer malfunctions, ineffective systems and records management

practices, and miscommunication between EOP components led to e-mail

records not being preserved by ARMS. As a result, the EOP initiated a

time-consuming and expensive effort to recover e-mail records that had

not been effectively managed.

Limited Access to

Information

Prevented

Confirmation of the

EOP’s Notification of

Appropriate Officials

Conclusions
Page 17 GAO-01-446 EOP E-mail System

The management control weaknesses have implications for the current

administration that inherited the core e-mail systems and records

management structures discussed in this report. Accordingly, we met with

the current Special Assistant to the President and Director of the Office of

Administration and interested EOP staff, to provide suggestions for

establishing effective controls in the records management and e-mail

system support functions. These officials stated that actions are underway

to improve the management of electronic records within EOP. These

efforts included developing and updating policies for federal and

presidential records, defining procedures for electronic records, and

establishing a chief information officer position.

In written comments on a draft of this report, which are reprinted in

appendix VI, the Counsel to former Vice President Gore had two general

points. First, he stated that the report did not explicitly state that we found

no evidence of attempts by OVP staff to deliberately fail to preserve

records. In our report, our objective was to examine the facts surrounding

the e-mail systems. Specifically, we were requested to assess the adequacy

of management controls over EOP’s records management system,

particularly as they related to the numerous e-mail system malfunctions

experienced by EOP since 1996. That assessment entailed the examination

of management policies, programs, and practices as they related to

effective records management controls and to the requirements of the PRA

and FRA. Our review was not directed at assessing the intent of

individuals responsible for preserving records and providing responsive

records to investigative bodies. As such, we can offer no conclusions on

this matter.

Second, the Office of the former Vice President stated that ARMS was a

unique system for which best practices did not apply and provided

additional information on the creation and use of this system that was not

reflected in the draft report. In response, we have revised the report to

reflect this additional information. However, we disagree with the Office

of the former Vice President’s characterization of the applicability of best

practices to ARMS. Best practices for management of records currently

exist in the form of National Archives and Records Administration

guidance contained in title 36 CFR and apply to records management

programs irrespective of the type of system involved. The majority of this

guidance is prescriptive only for federal records; however, this guidance

applies to ARMS because ARMS archives federal records. Also, the

guidance does not explicitly preclude application of these practices in the

management of presidential/vice presidential records. Although the

Agency Comments

and Our Evaluation
Page 18 GAO-01-446 EOP E-mail System

evolution of ARMS may continue, the stability of generally accepted

records management best practices establishes a control foundation for

any system, whether manual or automated.

Comments from the Office of the former Vice President are reprinted in

appendix VI. In addition, this appendix contains our detailed responses to

numerous technical comments that were provided.

In his comments on the draft report, the representative of former

President Clinton stated that he had reviewed the draft but did not have

access to the records necessary to properly review and comment on its

accuracy. He urged us to incorporate comments that we received from

EOP during the course of our review, which we have done as appropriate.

The Deputy General Counsel, Office of Administration, EOP, orally

provided a technical clarification, stating that the May 1997 policy that

replaced the PRA section of the 1993 policy was issued by the White

House and not EOP, which we incorporated as appropriate.

We are sending copies of this report to the Honorable Henry Waxman,

Ranking Minority Member; the Honorable Hector Irastorza, the Deputy

Assistant to the President for Management and Administration, the White

House; Bruce R. Lindsey, the personal representative of former President

Clinton; Andrew M. Wright, Counsel to former Vice President Gore; and

interested congressional committees. Copies will then be available on our

Web site at www.gao.gov.

If you or your office have any questions about this report, please contact

me at (202) 512-6240 or by e-mail at KoontzL@gao.gov. Nancy

DeFrancesco, Michael Fruitman, Linda Lambert, and Charles Roney were

major contributors to this report.

Sincerely yours,

Linda D. Koontz

Director, Information Management Issues

Enclosures
Appendix I: Scope and Methodology

Page 19 GAO-01-446 EOP E-mail System

To obtain information regarding events and to identify individuals key to

the Executive Office of the President’s (EOP) e-mail systems and the

malfunctions experienced, we

• reviewed about 10,000 pages of documentation submitted to the House

Committee on Government Reform by the EOP and its contractor

Northrop Grumman;

• reviewed 159 hearing exhibits prepared by the Committee from the above

documentation, and pertinent pieces of correspondence and other related

documentation provided by the Committee;

• obtained copies of the EOP’s task order OA8004 dated September 30, 1997,

with Northrop Grumman, and letter contract OA20C002 dated March 29,

2000, with Enterprise Computing Solutions Technologies, Incorporated

(ECS);

• obtained transcripts of a hearing pertaining to the timing of the restoration

and production of EOP e-mail, held by the U.S. District Court beginning

July 13, 2000, and observed the hearing sessions between July 31 and

October 3, 2000; and

• obtained transcripts of hearings pertaining the EOP e-mail systems, held

by the House Committee on Government Reform on March 23, March 30,

May 3, and May 4, 2000.

To obtain additional information on the e-mail malfunctions and Office of

the Vice President’s e-mail records procedures that were not addressed by

the Committee documents or hearing transcripts, the EOP stipulated that

we submit written questions. We submitted seven sets of questions and

reviewed the written responses and additional documentation provided,

including two additional tape restoration project contracts: Vistronix,

Incorporated, letter contract CI20C03 dated May 8, 2000; and TW and

Company contract DC21C01 dated November 9, 2000. We also met with

EOP officials to discuss specific issues pertaining to the restoration of the

e-mail.

To determine costs for the project, we obtained copies of paid ECS

invoices covering the period May 5 through September 5, 2000. We also

obtained a copy of the July 28, 2000, Vistronix report that estimates project

costs and compared this estimate to updated information regarding

project status and actual costs incurred for the project through October

29, 2000, that we obtained from the EOP.

Our review of cost issues was limited by the unavailability of complete and

reliable information. Specifically, we were not given access to OVP files of

hard copy e-mail records, which impaired a full assessment of the OVP’s

Appendix I: Scope and Methodology
Appendix I: Scope and Methodology

Page 20 GAO-01-446 EOP E-mail System

practices to preserve e-mail records. In addition, we were not provided

complete documentation of contract performance assessments and related

consequences for fiscal years 1998, 1999, and 2000; thus, we were unable

to determine the extent to which the EOP documented contractor

performance weaknesses and withheld award fees. Finally, we were not

provided with copies of letters from the EOP to investigative bodies

necessary for us to confirm that the EOP notified these entities of the email malfunctions.

We performed our work from July 2000 through January 2001, in

accordance with generally accepted government auditing standards.
Appendix II: Chronology of E-mail

Malfunctions

Page 21 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

E5346 through

E5347, E5353

through E5354

07/94 and

10/94

E-mail from Nell Doering to various staff

including Daniel Barry, James Wright, and

others establishing an e-mail monitoring

program that was implemented 07/25/94. The

Presidential and Office of the Vice President

(OVP) Records Monitoring Plan dated 10/28/94

that was attached provided a quarterly

monitoring schedule through fiscal year (FY)

1995.

This monitoring program

required that EOP

agencies review a sample

of record and nonrecord

e-mail messages to

ensure that proper record

designation was applied

by e-mail users.

E0595 through

E0608

03/07/95 A document entitled Automated Records

Management System (ARMS) for Lotus Notes

Mail provided a technical description of the

Notes/ARMS interface functionality. The

document stated that the interface copies all

internal and external e-mail and receipts to the

ARMS records management database. White

House Executive Office of the President (EOP)

e-mail users may elect to remove the record

designation of an outgoing mail message,

which they created, by selecting a nonrecord

command button. This option causes the

message to not be captured as an official

federal or presidential record to be archived in

the ARMS records management database.

E5346 through

E5347 and E5353

through E5354

Nonrecords were also

captured temporarily by

ARMS for sampling under

EOP’s records monitoring

program to detect and

correct improper

designation of e-mail

records.

E1115 08/29/96 The Mail2 server was created with the server

name of Mail2 and User ID of Mail2/EOP.

E0564

E0564 through

E0566

10/29/96 E-mail stating that a Planning Research

Corporation (PRC) contract employee installed

the Mail2 server, as well as performed initial

configuration of the Notes server. The

Notes/ARMS interface Name and Address

Book (NAB) was configured with five views.

Exhibit 125 and

document E1115

PRC was the predecessor

contractor to Northrop

Grumman (NG) at the

White House.

E2309 through

E2310

02/25/97

and

02/26/97

Series of e-mail between Daniel Barry and

Laura Crabtree regarding “... ‘problem

attachment’ problem” whereby messages were

deleted manually if they caused a problem with

the Notes/ARMS interface.

Based on available

documentation, this

problem was not related

to the Mail2 or Letter D

malfunctions.

Exhibit 17 03/04/97 E-mail from Daniel Barry to John W. McGinnis

asking him to write a paragraph regarding

efforts performed to restore 10,000+ lost E-mail

records. Mr. McGinnis’ contribution would finish

up Mr. Barry’s “memo to the record.”

The 3/23/00

hearing witness

list lists Mr. Barry

as Daniel “Tony”

Barry. He was

referred to as

Tony in many emails and other

documents.

Based on available

documentation, this

problem was not related

to the Mail2 or Letter D

malfunctions.

Appendix II: Chronology of E-mail

Malfunctions
Appendix II: Chronology of E-mail

Malfunctions

Page 22 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

E2316 through

E2323

03/06/97 Series of e-mails from Daniel Barry to James

Wright and others regarding “missing Notes

records.” The problem occurred in September

1996 and caused some e-mail to be sent to the

ARMS records management database without

the message header. Other e-mails would be

sent to the ARMS records management

database without the message text. Mr. Barry

stated he had received no training on Notes. He

reported that to his knowledge, efforts to

recover these missing e-mail messages in the

beginning of 1997 were unsuccessful.

According to this e-mail, between September

11 and 16, 1996 23,436 e-mail messages were

missing headers (that is, “To,” “Subject,” etc.)

and 10,138 messages were missing text.

Exhibit 17 Based on available

documentation, this

problem was not related

to the Mail2 or Letter D

malfunctions.

E6001 through

E6004

03/17/97 Minutes of a staff meeting involving Jim Wright,

Laura Crabtree, Karl Heissner, and others.

Backup of EOP Information Systems and

Technology Division (IS&T) servers was

discussed. The minutes stated that

“[r]equirements haven’t been documented but

approaches are being developed. An inventory

does exist of what does need to be backed up.”

The minutes also question what the retention

period should be, and stated that all types of

server platforms “are being reviewed: VAX,

client server, and mainframe.”

E6547 through

E6558

03/21/97 E-mail to Ada Posey and others regarding

restricted 1997 appropriation funding and that

funds were needed for additional e-mail servers

to relieve network strain. E-mail records

management traffic was projected to increase

90 percent in FY 1998 from FY 1995’s traffic

rates. The e-mail noted that EOP’s migration

from Novell Netware-based network to

Microsoft Windows NT began in May 1996. It

also stated that the “...client/server backup

system used to preserve data integrity and

facilitate the archival of Presidential and

Federal records has exceeded its life cycle [and

can only] backup 50% of our file servers.

However, due to hardware and software

failures, less than 30% of our file servers have

current backups. Currently, the only copy of

data that is not backed up resides on each file

server’s physical hard drives. If any of those

hard drives fail, data will be lost.”

E1022, E1394,

Exhibit 9
Appendix II: Chronology of E-mail

Malfunctions

Page 23 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

E6005 through

E6009

03/24/97 Minutes of a staff meeting involving Laura

Crabtree, Nell Doering, Karl Heissner, Jim

Wright, and others. The minutes discussed

external e-mail, stating “Laura will have a report

on the problems we’ve been having for Ada

[Posey] tomorrow.”

Based on available

documentation, these

problems were not related

to the Mail2 or Letter D

malfunctions.

E2379 through

E2380

07/17/97 E-mail from Bruce Overton to Laura Crabtree

and others with attached e-mail notice to all

EOP mail users regarding implementation of

Notes janitor agent. The janitor agent was set

to move up to 300 e-mail messages that were

greater than 45 days old from the “inbox” folder

to the “trash” folder in the user’s mail file. The

messages were then automatically deleted 1

week later. The process was scheduled to run

weekly beginning July 18, 1997. Mr. Overton

asked whether the records management team

had reviewed the janitor policy.

A 9/29/00 response to our

question on this matter

stated that “.... EOP does

not believe that this

‘janitor agent’ was ever

implemented.” In its

comments on a draft of

this report, EOP stated

that “... legal and records

management staff never

approved use of the

agent.”

E7209 through

E7211

08/11/97 E-mail to Daniel Barry, Nellie Doering, Karl

Heissner, Laura Crabtree, Ada Posey, and

others containing minutes of a staff meeting.

The minutes state that “Lotus Notes staff will be

[coming] in to address the lost mail problem (we

need a ruling from [counsel] as to issues with

mail and as to what may be purchased).”

Based on available

documentation, this

problem was not related

to the Mail2 or Letter D

malfunctions.

Exhibit 49 01/30/98 A document created by Daniel Barry said that

he noticed a potential problem with the capture

of e-mail traffic coming into the EOP Lotus

Notes system from the Internet. John Spriggs,

the NG e-mail administrator, was asked to print

out the firewall log to determine whether

incoming Internet e-mail messages were

received. The document stated “We are not

sure where the problem lies at this point.”

Exhibit 57 A computer directory

location and date

indicating 1/30/98 were

handwritten on the

document.

Exhibit 10 No date A document containing May 16 meeting

minutes discusses a problem with the

Notes/ARMS interface scan process randomly

skipping e-mail boxes on the Mail2 server. A

random sample of e-mail boxes was analyzed

and it was discovered that e-mails dating back

to 1996 were not subject to records

management, or captured by the ARMS

records management database. The course of

action was to further assess the problem.

Exhibit 12
Appendix II: Chronology of E-mail

Malfunctions

Page 24 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

Exhibit 16 06/01/98 E-mail from Virginia Apuzzo to all EOP staff

requesting that users delete unneeded e-mail

on e-mail servers or save e-mail to individual

workstations to prevent system failure. This

message was repeated and re-sent to all EOP

staff on 03/24/99 (Exhibit 16, page E0545).

E7202 through

E7203

Exhibit 12 06/12/98 E-mail from Robert Haas, NG contract

employee, to Betty Lambuth, NG subcontractor,

mentioning “a design flow in the Record

Management design in Notes that would allow

for certain types of mail document[s] to bypass

record management.”

Exhibit 10 The term “design flow”

may be a typo of “design

flaw”.

Exhibit 50 06/18/98 A draft document entitled Lotus Notes to ARMS

Interface Anomaly that was sent from the fax

machine shared by the Office of Administration

(OA) General Counsel and the OA Director to

an unknown recipient. The draft described the

Mail2 server malfunction and stated that the

root of the problem stems from a discrepancy

between the capitalization of the letters in the

name of the affected Lotus Notes server. The

problem was introduced by human error in

capitalization of the server name letters in the

affected users’ e-mail accounts as “MAIL2”

instead of “Mail2.” The scope of the problem

was that 526 users of Mail2 were affected.

Exhibits 1 through 5 06/19/98 Memo from Virginia M. Apuzzo to John D.

Podesta, former Deputy Chief of Staff, entitled

Technical Anomaly in Automated E-Mail

Records Management System, which describes

the capitalization of the server name letters in

the affected users’ e-mail accounts as “MAIL2”

instead of “Mail2” as the cause of the Mail2

malfunction. Exhibits 1 and 3 contained a

handwritten note saying, “Chuck—I sent this

memo to John this afternoon. Ginny” dated

6/19. Exhibits 4 and 5 contained a handwritten

note saying, “Ginny Please ask Mark to brief

me on this. Thanks, John.”

Exhibits 1 through 5 were

variations of the same

memo. U.S. District Court

testimony by Charles Ruff

on 08/28/00 confirmed

“Chuck” to be Charles

Ruff, “Ginny” to be

Virginia Apuzzo, and

“John” to be John

Podesta.

E0567 through

E0569, E0573

through E0578,

E0589 through

E0594

07/09/98 Draft plan dated July 1998 and the undated

finalized versions of the same plan to first

process data records from Notes e-mail files to

the VAX subsystem and then to restore

Notes/ARMS interface processing on Mail2.

The plan was to first turn the Notes/ARMS

interface monitor task off, copy each affected

user’s existing mail file (containing the

unrecorded mail) to the Notes/ARMS interface

server, and then turn the Notes/ARMS interface

Exhibit 19
Appendix II: Chronology of E-mail

Malfunctions

Page 25 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

monitor back on for processing of unrecorded

messages into the ARMS records management

database. Once transferred, all documents

would be marked as recorded and the syntax

[capitalization of the server name letters in the

affected users’ e-mail accounts as “MAIL2”

instead of “Mail2”] of the server name for the

user’s account would be corrected. Day-forward

processing would resume on Mail2. The plan

also noted that a “critical issue” existed for the

processing of affected records from Notes to

the ARMS records management database,

which resides on the VAX subsystem. It stated

that “preliminary investigation has shown that

the date used to create the ARMS header files

is the date that the messages are written to the

Records Management mail-in database. This

date doesn’t reflect the true sent mail date in

the original message.”

Exhibit 19 07/24/98 E-mail from Daniel Barry to James Wright

noting under Additional activities that he

continues to be involved in discussions

regarding the Mail2 problem but there has been

no movement thus far in correcting the problem

or getting the data over to the ARMS records

management database. The plan for fixing the

problem had been submitted to EOP by NG.

Exhibit 22 07/30/98 E-mail from Daniel Barry to James Wright with

language similar to Exhibit 19.

Exhibit 23 08/13/98 E-mail from Daniel Barry to James Wright

expressing “concerns” about “the mail2

problem” or project X. He stated that there was

no movement underway to fix the problem and

recover the lost records from the backup tapes.

Exhibit 24

Exhibit 24 08/13/98 E-mail from James Wright to Daniel Barry

responding to his “concerns” e-mail saying that

there has been some movement to get back on

track and NG can develop a plan to get this

effort going. “Certainly the Data Center and the

Records Team has been left out of this matter

and the result could be a great deal of work put

upon us later.” This exhibit stated that Kathleen

Gallant informed Mr. Wright that a Paulette

[Cichon] briefed Jim Welsh of NG to proceed

with developing the plan.

Exhibit 23
Appendix II: Chronology of E-mail

Malfunctions

Page 26 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

Exhibit 40 08/13/98 E-mail from Kathleen Gallant to James Wright

with a copy to Daniel Barry. It stated that Jim

Webster was taking Betty Lambuth’s place and

a meeting was held authorizing that it was okay

to discuss the Mail2 project in detail with Mr.

Webster. The e-mail stated that “I also agree

with Tony [Daniel Barry] about the new

searches that will have to be done. We need

direction from OA counsel on that front.”

Exhibit 25 09/01/98 E-mail from Daniel Barry to James Webster,

with a copy to James Wright, asking for a

meeting to discuss the plan/approach on the

Mail2 problem. An attached e-mail “Weekly”

report dated 9/4/98 (E2016) says that Mr. Barry

had a discussion with Jim Webster, NG,

“regarding OVP E-mail and records

management.” Mr. Barry referred Mr. Webster

to Kathleen Gallant for a decision to have all

OVP users subject to records management

through the ARMS records management

database. The e-mail noted that a goods and

services authorization request memo was

prepared for 64GB disks for the Notes Mail2

recovery project. This will go to Financial

Management Division (FMD) for signature and

then make its way to the OA counsel.

Exhibit 120 09/01/98,

09/15/98

E-mail from Daniel Barry to James Webster,

NG, requesting meeting with him, Sandra

Golas, Robert Haas, and John Spriggs to

discuss the Mail2 problem and the

plan/approach for proceeding. Webster

forwarded Barry’s e-mail to Spriggs, Golas,

Robert Haas, and Yiman Salim, saying Barry

wants “to set up a meeting to discuss our

favorite issue.... I will tell Tony [Daniel Barry]

that I will have to delay any meeting until we

resolve some internal issues.” On page E4015

of this exhibit, an e-mail dated 9/15/98 from

Daniel Barry to James Webster about the Mail2

project meeting asks about Webster’s plan for

“righting the wrong” phase of this project. Barry

wants to know when the meeting will occur.

E5874 09/10/98 E-mail from Mark Bartholomew to Adam

Greenstone [copy to Daniel Barry, Karl

Heissner, Kathleen Gallant] regarding

processing of incoming Internet e-mail to the

President, Vice President, and the First Lady,

referred to as White House “principals.”

Incoming e-mail to these principals was

E6020 through

E6022

The bulk e-mail “package”

was captured in the

ARMS records

management database.
Appendix II: Chronology of E-mail

Malfunctions

Page 27 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

“packaged” into another e-mail to the

correspondence office for each respective

principal. These “packaged” e-mails, referred to

as “bulk mail,” were not subject to records

management in the ARMS records

management database message by message

or as e-mail to the principal, but rather as bulk

e-mail to the appropriate representative of the

principal.

Exhibit 26 09/10/98 E-mail from Daniel Barry to Kathleen Gallant

and James Wright expressing concern about

the lack of movement on the Mail2 problem.

“We have known about this problem for 4

months now and not a single record [of the

affected users] has been passed to

ARMS...even worse, the root problem has not

been fixed.”

Exhibit 64 09/14/98 Letter from Joseph Lucente noting “that in late

May of this year, a dysfunction in the EOP email system was detected by an employee...” of

NG.

Exhibit 27 09/25/98 E-mail from Daniel Barry to James Wright

stating there was still no movement in the Mail2

problem and asking what his role was

supposed to be in the project. Wright responds

the same day saying that IS&T needs to start

by providing NG with direction.

Exhibit 47 09/25/98 E-mail from Kathleen Gallant to Daniel Barry

stating that the Notes Anomaly Team has met

and come up with a strategy. The team will

back up Mail2 using a DLT [digital linear tape]

drive that can verify as it writes and execute

“the Notes agent changes to change all the

[IDs] looking for Mail2 to MAIL2....” A second

phase deals with reconstruction of the

nonrecords-managed files from the Mail2

server. She said, “Contracts is aware of the

whole mess, and supports the creation of the

IWO [Internal Work Order]....”

E1235 and E1236 Agents are filtering

instructions that automate

operations on documents

in a Notes database.

E1363 through

E1375

09/30/98 A NG document entitled Final Y2K Status

Report for OA/IS&T stated that the software for

the ARMS records management database was

not fully documented and controlled, and that it

was “born very quickly in 1994 to meet an

immediate need for the recording of message

traffic.” The C-language programs in the

interface were not Year 2000-compliant.

E0766 and E1015

through E1021

Year 2000 readiness of

information systems was

a concurrent effort to

repair e-mail

malfunctions.
Appendix II: Chronology of E-mail

Malfunctions

Page 28 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

Exhibit 28 10/13/98 E-mail from Daniel Barry to Christina

VanFossan and Joseph Kouba stating, “Per our

meeting last week... here are 4 projects that I

can see happening in the near term that are not

currently budgeted for.” The third project was

listed as Project X (Mail2) Reconstruction, with

an estimated cost of $250,000.

E0766 through

E0768

11/06/98,

11/12/98

E-mail created by Marvin Miller on 11/06/98

and revised 11/12/98 by Yiman Salim regarding

an ARMS IWO feasibility and requirements

study. The document stated that Notes and

Notes Application Programming Interface (API)

version 4.6 software was used in the

Notes/ARMS interface. To perform

maintenance on the ARMS software, the skill

set required a C++ software engineer, a Notes

developer with C++ and VAX knowledge, and a

system administrator with Windows NT and

Lotus Notes skills. The tasks listed in the emails included Year 2000 code modification

and implementation of calendaring and

scheduling modifications. However, in the

11/12/00 revision, Salim removed two tasks

from Miller’s list, including “Mail server name

case sensitivity.”

E1363 through

E1375

“Name case sensitivity”

refers to the capitalization

of the server name letters

in the affected users’ email accounts as “MAIL2”

instead of “Mail2.”

In its comments on a draft

of this report, EOP stated

that the Mail2 case

sensitivity was not related

to Year 2000 efforts

addressed by the “ARMS

IWO.”

Exhibit 30 11/12/98 E-mail from Daniel Barry to DeVere Patton

saying that NG needs technical guidance on

several items, including a response to the Mail2

IWO, which had not yet been finalized.

Exhibits 31 and

46

Exhibit 31 11/13/98 E-mail from Daniel Barry to James Wright

mentioning the response on the Mail2 IWO.

Exhibits 30 and

46

Exhibit 121 11/16/98 E-mail from Daniel Barry to Sheryl Hall, John

Spriggs, and Joseph Vasta requesting a

meeting to discuss a course of action on

moving the Mail2 reconstruction project

forward.

Exhibit 103 11/20/98 E-mail from Daniel Barry stating that Karl

Heissner was the project manager for phases

two and three of the Mail2 reconstruction work.

The e-mail mentioned that phase one was the

“.... fix (Stop the bleeding) on the mail 2

server.” Phases two and three were not

described in this e-mail.
Appendix II: Chronology of E-mail

Malfunctions

Page 29 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

Exhibit 70 12/02/98 Weekly Contracting Officer’s Technical

Representative (COTR) meeting minutes for

11/20/98. NG commented that the Rough Order

of Magnitude (ROM) “...could be plus or minus

20 percent”

Exhibits 63, 67,

72, 73, 75, and

122

Exhibit 72 12/02/98 NG ROM for cost of IWO was $602,492. Exhibits 63, 67,

70, 73, 75, 122,

E1016

Exhibit 122 12/02/98 E-mail from Daniel Barry to Karl Heissner

stating that NG’s ROM did not include the cost

to do the restoration.

Exhibits 63, 67,

70, 72, 73, and 75

E5783 12/04/98 E-mail from Moe Vela to “All Staff” requesting

staff to delete unnecessary e-mail because

“Lotus Notes is within 5 [percent] of maximum

capacity and it will shut down at that time.”

Exhibit 63 12/11/98 Memo from Joseph Vasta to DeVere Patton,

mentioning, in part, the Mail2 E-mail

Reconstruction. Said NG would evaluate at

least one tape.

Exhibits 67, 70,

72, 73, 75, and

122

Exhibit 73 12/18/98 Weekly COTR meeting minutes for 12/16/98.

“Government emphasized that before any work

be undertaken, it is imperative that a tape

inventory be done.” NG insisted that an

inventory in the absence of an automated tape

management system would be very laborintensive (over 1 hour per tape) and “would far

exceed the services in the Base Services IWO.”

Exhibits 63, 67,

70, 72, 75, and

122

Exhibit 67 12/23/98 Weekly COTR meeting minutes for 12/23/98.

NG reported that a recent tape was restored,

and the universe of tapes was identified as 788.

Exhibits 63, 70,

72, 73, and 122

Exhibit 32 12/24/98 E-mail from Daniel Barry to Joseph Kouba. It

lists three tiers of projects related to the

decisions resulting from the Armstrong court

case. The third tier includes projects that, in his

opinion, could be done but were not vital. The

Mail2 reconstruction (Project X) was listed in

the third tier, showing an FY 1999 cost of

$650,000 and an FY 2000 cost of $1 million.

The Armstrong court held

that EOP components

must capture electronic

records subject to the

Federal Records Act.

Exhibit 129 01/06/99 Weekly report from Daniel Barry in which he

stressed that OA Counsel needed to be in the

decision-making process regarding projects

that should move forward, such as Project X.

E6020 through

E6022

01/07/99 Meeting agenda and handwritten meeting notes

regarding “records disposition for internet email.” It stated that for e-mail to “... ‘President

@whitehouse.gov,’ all emails [were] combined

into one and sent to [the President’s bulk e-mail

E5874
Appendix II: Chronology of E-mail

Malfunctions

Page 30 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

representative].” It was packaged and sent on

to the ARMS records management database

where it was retained as e-mail to the

President’s bulk e-mail representative.

E5784 01/13/99 E-mail from Moe Vela to “All Staff” requesting

staff to delete unnecessary e-mail and

databases in Lotus Notes because the IS&T

anticipated running out of disk space in about 1

hour.

E0772 02/01/99 E-mail from Daniel Barry about “the effort

underway to locate and document the

[Notes/ARMS] interface code so that changes

can be made.”

E0964 02/01/99 E-mail from Daniel Barry to Raul Cavazos

stating that “.... the ARMS processing

requested a tape from March 1997 and Ops

[Operations] was unable to find the tape.”

Concludes, “there needs to be an inventory of

the ARMS tapes that are stored in the boxes in

the back of the DC [data center], to make it

possible to retrieve any one of the tapes.”

Exhibit 75 02/02/99 Weekly COTR meeting minutes for 01/27/99.

The EOP asked about NG’s tape inventory and

tape tracking methodology. The EOP requested

a tape inventory (that included tape ID, volume,

dataset ID, etc.) from October 1997, the point

when NG was awarded contract.

Exhibits 63, 67,

70, 72, 73, and

122

Exhibit 81 02/05/99 Memo from Karl Heissner explaining the Mail2

server problem. He stated that “the problem,

that some e-mail messages were not records

managed properly, was discovered in February

1998, and Mr. Barry immediately notified the

Lotus Notes Group (NG-Spriggs/Golas) as well

as IS&T Management of the problem.”

Exhibits 82 and

83

Exhibit 43 03/12/99 E-mail from Nellie Doering to Dorothy Cleal,

copy to Daniel Barry, suggesting changes to

Mr. Barry’s proposal. Ms. Doering stated that

Mr. Barry agreed to ensure the “Server 2” tapes

were properly inventoried and documented by

NG.

Exhibit 37 03/18/99 E-mail from Daniel Barry to DeVere Patton, Karl

Heissner, and Nellie Doering about the Mail2

records management problem. Barry stated

that: “It has come to my attention that when the

‘bleeding’ was stopped on MAIL2 in November

1998, ALL the bleeding may NOT have been

stopped. I have spoken with John Spriggs and

it appears as though at least one account,
Appendix II: Chronology of E-mail

Malfunctions

Page 31 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

MILLENNIUM, may still have the problem. I

believe NG should be instructed to investigate

and report back on exactly what the situation is

with regard to the MAIL2 problem.”

Exhibit 41 03/18/99 A series of e-mails noting that the Mail2

problem was not allowing EOP staff to recycle

backup tapes. Nellie Doering stated that the

tapes need to be inventoried by NG and “....

‘secured’ so that they are easily identified as

the Server 2 Backup Tapes that were not

[subject to records management.]” DeVere

Patton stated that “...it is something that NG

should be doing under the base contract [in

response to Dorothy Cleal’s inquiry regarding

contract scope].

Exhibit 42 03/18/99 E-mail from Joseph Kouba to Daniel Barry and

others stating, “looks like MAIL 2 reconstruction

is back on hold until some additional

confirmation is received.”

Exhibit 124 03/19/99 E-mail from Karl Heissner to Nellie Doering,

Daniel Barry, and Mark Bartholomew. Meeting

minutes to discuss the Mail2 reconstruction. It

stated that the project was on hold awaiting

approval of the OA counsel.

E0812 03/24/99 E-mail from Michael Ritter to Joseph Vasta

regarding tape inventory. It stated that “there

are between 700 and 800 tapes [in] mostly 8-

mm [millimeter] format and do not have a

‘header’ catalogue (meaning that the entire

tape must be read to develop a directory list).”

Also, “there is a short period of a few months

where we do not have backup tapes due to

changes in the governments requirements.” Mr.

Ritter estimates that under “perfect conditions”

it would take 1.5 to 3 hours per tape to read the

tape and print the catalogue information;

however, he stated, the tapes were old and

were produced under various hardware and

software configurations. He stated that “it is

expected that there will be tapes that cannot

even be read with the current hardware on

site.”

E1255 through

E1256

03/25/99 NG memo from Joe Vasta to DeVere Patton

regarding Mail2 backup tapes. It stated that NG

will secure Mail2 backups for October 1996

through November 1998 and provide minimal

inventory information. The memo identified that

during this period, NG was instructed by “the

government” to reuse backup tapes causing
Appendix II: Chronology of E-mail

Malfunctions

Page 32 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

some to be overwritten. The memo stated that

a report detailing a description of the system

failure was beyond the scope of the base

contract.

Exhibit 117 04/01/99 E-mail from Michael Ritter to Yiman Salim,

Robert Haas, John Spriggs, and Sandra Golas

asking them a series of questions about when

the Mail2 server problem started and when it

was resolved. He requested responses from

each by 4/2/99.

E0833 and E0834

E1194 04/01/99 E-mail from Benjamin Kirby to Adam

Greenstone, Daniel Barry, others, stating the

millennium account was not subject to records

management.

Exhibit 66 04/02/99 E-mail from Yiman Salim to Michael Ritter

about the Mail2 issue. She stated that NG

personnel became aware of the Mail2 problem

on June 12, 1998, and that the cause of the

problem, writing Mail2 as MAIL2, was identified

a couple of days later by John Spriggs. She

also noted that the problem was inherent in the

ARMS process because it “is currently

enforcing server name case sensitivity.” The

message also stated that in December 1998

NG initiated a task to baseline the Notes/ARMS

interface program. Completion was scheduled

for April/May 1999, after which programming

changes could be made as needed.

Exhibit 117

Exhibit 71 04/02/99 E-mail from Michael Ritter to Joseph Vasta.

This e-mail explains the Mail2 server problem in

detail. It also mentions that NG staff have

identified a similar problem on MAIL1 that was

also caused by human error. The message

stated that an employee of the contractor PRC

designed a program to migrate user accounts

to the Notes server, and this program entered

the users with the improper server name

syntax.

Exhibit 44 04/09/99 Meeting notice from Karl Heissner about

records management of all Mail2 and MAIL1

server accounts. This notice stated that Mail2

and MAIL1 servers still contain

“unrecordsmanaged” accounts, preventing the

recycling of backup tapes. The objective was to

make necessary corrections by April 18.
Appendix II: Chronology of E-mail

Malfunctions

Page 33 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

Exhibit 125 04/09/99 Series of e-mails describing Letter D

malfunction. Marvin Miller e-mails to Robert

Whiteman and Yiman Salim that the Letter D

malfunction was caused in 11/98 when NG was

correcting a Lotus Notes buffer overflow system

failure. The user accounts (sorted by first

initial/last name) were distributed among five

Notes/ARMS interface e-mail server views, so

that all 26 letters of the alphabet were assigned

to one of the 5 views. “The letter ‘D’ was

inadvertently omitted and the letter ‘J’ was

added twice.” This caused all users on all email servers whose first names began with D to

not be subject to records management.

Exhibits 13, 79,

documents

E3218, E3219

through E3221,

and E3222

through E3229

E3219 through

E3221

04/15/99 Memo from Robert Whiteman (NG) to Dorothy

Cleal describing events regarding the Letter D

malfunction. It stated that a Notes/ARMS

interface e-mail server system failure occurred

in November 1998, which was caused by a

configuration error. In correcting the failure, NG

reconfigured the Notes/ARMS interface views

on Lotus Notes e-mail servers and

“inadvertently omitted” the letter D from the

views, causing incoming non-Notes e-mail for

users with first names beginning with D not to

be subject to records management. According

to the memo, the Lotus Notes team reported

this to NG management on 04/09/99. Mr.

Whiteman recommends not recycling tapes

until the problem was resolved and to include

the saved tapes in the ongoing e-mail recovery

effort.

Exhibit 79,

documents

E3218, and

E3222 through

E3229

E3218 04/15/99 Memo from Dorothy Cleal, through Michael

Lyle, to Mark Lindsay regarding the Letter D.

The memo stated that 191 users were affected

and who have not had e-mail records-managed

since November 1998. Cautions that “...records

management problem may not be limited to this

particular error.”

Exhibits 13 and

79; documents

E3218, E3219

through E3221,

and E3222

through E3229

Exhibit 79, E3222 04/15/99 Cover memo from Eric Ritter to Dorothy Cleal

with attached listing that shows the 191 users

impacted by the Letter D malfunction—54 on

MAIL1, 77 on Mail2, 29 on MAIL3, 30 on

MAIL5, and 1 on RDS1.

Exhibits 13, 125;

documents

E3218, E3219

through 3221, and

E3222 through

E3229

Exhibit 99 04/20/99 This e-mail from Joseph Kouba stated that a

“legal determination [was needed] that

reconstruction is required by the court case and

that this is a legitimate use of Armstrong funds.”
Appendix II: Chronology of E-mail

Malfunctions

Page 34 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

E7265 through

E7268

04/28/99 E-mail from Robert Whiteman to Marvin Miller,

Robert Haas, Yiman Salim and others

requesting review of Lotus Notes self

evaluation for 10/98 to 04/99. It stated that

Notes staff began analysis phase of the

Notes/ARMS interface task. During this

analysis, staff identified a records management

problem (Letter D malfunction) and were

researching viable solutions.

E1235 and E1236 05/17/99 Report explaining e-mail audit agents that run

on Notes servers and search for unrecorded

mail and incorrect mail templates. It

recommends running audit agents weekly and

providing logs to Mr. Barry. It also recommends

resuming tape recycling. It stated that “it would

not be possible to recycle any back-up tapes

that were created prior to the (managed date)

completion of the correction and verification of

the system...”.

Exhibit 47

Exhibit 38 05/21/99 E-mail from Michael Ritter to Albert Leister

regaining the Mail2 Statement of Work (SOW).

“This is actually from September ’98. Still no

action on...or response from...the gov’t.”

Exhibit 118 05/21/99 E-mail from Albert Leister Jr. to Dorothy Cleal

stating that the “...backup tapes are in the DC

[data center].”

E0793 05/21/99 E-mail from Marvin Miller showing a schedule

to run various Notes agents. One scheduled for

05/25/99 was to “fix letter ‘D’ and letter case

problem. Records will be managed from this

point on.”

E4483 06/04/99 E-mail from Albert Leister to Dorothy Cleal

reporting that the Notes audit agents indicate all

documents requiring records management

were managed properly and the agents run

weekly “.... to insure no new errors surface.”

The e-mail noted that audit logs were given to

Karl Heissner and requests personal direction

to resume tape recycling.

E0798 06/09/99 E-mail from Yiman Salim stating that “...the

[Notes/ARMS interface] views, in the Name and

Address Book, need to be modified to make the

selection criteria non-case sensitive.” The

Notes/ARMS interface scanning process had

skipped a user account (that was not in use)

because the name started with lower case. “A

CMC [configuration management change] has

been submitted to implement this modification.”
Appendix II: Chronology of E-mail

Malfunctions

Page 35 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

Exhibit 6 01/13/00 Briefing materials for 1/19/00 meeting with the

National Archives and Records Administration

regarding e-mail issues. It mentions Mail2 and

Letter D configuration issues. It stated that

Mail2 affected 526 users and Letter D affected

about 200 users.

Exhibit 130 01/21/00 Chronology of Mail2 server events and impact,

labeled “Tony” in upper right-hand corner. It

discusses the problem and plan of action. It

also identified the magnitude of e-mails being

researched as encompassing 850 tapes, and

that some tapes contain classified data. The

plan of action says that Enterprise Computing

Solutions Technology, Incorporated (ECS), an

8a firm, has prior experience and was currently

on-site for the Armstrong effort.

E4337 through

E4338

02/01/00 E-mail from Daniel Barry to Michael Sullivan

with comments on an attached draft request for

proposals (RFP) for NG to bid on recovery of

lost e-mails. Barry comments that a

subcontractor to Information Management

Consultants, Incorporated (IMC), had

developed the original Notes/ARMS interface

code in 1996. The e-mail stated that the

Notes/ARMS interface transitioned to NG in

1997/1998. The system broke in late 1998

because it could not handle increased user

population when migrating from All-in-1 [the

previous e-mail system] to Notes, causing the

Letter D problem. The RFP requests a proposal

from NG by 02/15/00, and stated that NG

caused the problem and that “...corrective

action should be at no-cost to the Government.”

Barry asks if it was “fair and equitable” to hold

NG accountable for fixing bugs in the software

developed by the IMC subcontractor.

E5978 through

E5979

E1393 02/25/00 Handwritten notes stating that some tapes

included in the tape recovery effort were

classified as confidential or secret.

E1394 02/25/00 Handwritten notes of an interview about “Bob’s

– Mail issue,” which state that backup problems

were experienced due to going from a File

Allocation Table (FAT) system to a Windows

NT File System (NTFS). An accompanying

timeline shows the backup problem existed

from the beginning of the Mail2 problem (10/96)

until about 11/97.

Exhibit 9, E1022,

and E6552
Appendix II: Chronology of E-mail

Malfunctions

Page 36 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

E3237 No date Document entitled Tape Reconstruction

Procedure Outline stated that 850 tapes were

involved in the recovery effort.

E4349 through

E4354

02/29/00 Ontrack Computer Evidence Services’

Statement of Work and Proposal for Services

proposes a cost of $436,000 plus expenses

and 55 days to copy and catalog 800 tapes at

the EOP, which would require acquisition of 11

computers. It would cost $348,000 and 55 days

to perform the work at Ontrack’s facility, which

was fully equipped. Pricing for data recovery

varied depending on four categories of optional

services.

E3247 through

E3248

03/01/00 E-mails between John Spriggs, Albert Leister,

Terrence Misich, and William Burkey regarding

backup tapes. Magnitude of tapes revised by

Burkey to 3,391, including 283 “miscellaneous

tapes.” Spriggs stated that “we really do not

have any good evidence of what backup

[device] was actually backing up Mail2 on any

given date.” Leister identified three boxes of

tapes containing 1,006 8-mm tapes and 51

DLTtapes as definite to search and identified

another three boxes containing 503 8-mm

tapes as “possible,” for a total of 1,560 tapes.

The tapes consist of 4-mm, 8-mm, and

DLTtape media spanning intermittent periods

from 1/96 to 6/99. The tapes reportedly

contained backup of Office of Management and

Budget (OMB), White House Office, and other

EOP agency servers, including Mail2.

Exhibit 46 No date Timeline for ARMS-related activities

(Development Tasks, Problem Discovery and

Problem Fixing). It lists the Mail2 problem as

discovered on 6/12/98 and fixed on 11/22/98.

The Letter “D” was problem discovered on

4/9/99 and fixed on 6/1/99.

Exhibit 9 No date Draft Talking Points on the Mail2 Server

Anomaly stated that incoming e-mails exist on

backup tapes, “...except for a small window of

time from 8/96—11/97 when, due to a glitch in

the backup software, the mail files may not

have been backed up on a consistent basis.”

E1022, E1394,

and E6552

E1040 through

E1041

03/10/00 Memo from Terrence Misich to Jack Young,

General Counsel/OA. It stated that he “visually

E1015 through

E1021
Appendix II: Chronology of E-mail

Malfunctions

Page 37 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

scanned 14 boxes of magnetic media,

containing 3,391 magnetic tapes of various

types” and that a number of tapes do not have

external labels and some may have erroneous

external labels. Mr. Misich recommends the

performance of the first phase of the Tape

Restoration Plan.

Exhibit 13 03/14/00 Cover note from Yiman Salim with attached

audit log files from May 8 and 9, 1999, for EOP

mail servers Mail2 and MAIL1 showing

unrecorded documents. According to the logs,

in May 1999, both Mail2 and MAIL1 had four

user accounts each with server name syntax

errors causing unrecorded mail dating back to

1997. The logs also show 126 users affected by

the “Letter D” malfunction (70 on Mail2 and 56

on MAIL1).

Exhibits 37, 44,

71, 79 and 125.

Logs show that about 5

months after the Mail2 fix

was implemented, Mail2

still had four user

accounts with syntax

errors and unrecorded

mail dating back to 1997-

1998. Also, MAIL1 had

four user accounts with

syntax errors in this same

time frame.

E1015 through

E1021

03/15/00 A document entitled Draft Tape Restoration

Plan revised to include new data from ECS. It

stated that 3,391 tapes were involved in the

recovery effort and attributes the Letter D

malfunction to a “...software failure.” It also

attributes the Mail2 malfunction to case

sensitivity. It stated that “work associated with

this data recovery began almost as soon as the

problem was discovered.” The draft plan

mentioned that the work was not begun in

October 1998 when the SOW was developed

for NG because of Year 2000 efforts, so the

recovery was postponed to March 2000. In

February 2000, OA “...began exploring

alternatives,” and received an estimate from

one company of $436,000 and 55 days to

extract mail from 850 tapes. The draft plan

extended these estimates to $1,744,000 and

220 days for 3,391 tapes. The draft plan also

stated that another vendor, familiar with ARMS,

estimated $3,286,596 and 1.7 years.

Exhibit 72, E1363

through E1375,

and E0776

E1022 No date Hand-drawn timeline of e-mail malfunctions.

Shows October 1996 to October 1997 “backup

tapes missing, FAT to NSF file...Fixed.” August

1998 “ARMS research to fix Mail2 – discover

they cannot.” November 20, 1998, Mail2 stop

bleed, tape recycling starts November 21. April

9, 1999, D problem identified, stopped tape

recycle. May 21, 1999, “stopped D-bleed”.

E1394, E6552,

and Exhibit 9

In its comments on a draft

of this report, EOP stated

that “.... tapes were not

recycled from June of

1998 until June of 1999.”
Appendix II: Chronology of E-mail

Malfunctions

Page 38 GAO-01-446 EOP E-mail System

Referencea

documents Date Description of eventa

Cross-reference

to other

documents

listed in this

tablea Commentsa

Exhibit 57 03/17/00 Fax from Beth Nolan, White House Counsel, to

The Honorable Dan Burton, stated, “In January

1998, Daniel Barry....noticed a possible

anomaly within ARMS...that some incoming emails might be missing....Mr. Barry notified his

superiors and documented his finding...The full

extent of the error causing the anomaly Mr.

Barry noted was not discovered until June

1998, when NG employees

discovered....certain incoming e-mail

messages that were coded as

“unrecorded”....The contractor notified IS&T

personnel....By the fall of 1998, NG technical

personnel working with IS&T staff discovered

that the problem was due to miscoding ‘Mail2’

as ‘MAIL2.’ They further determined that the

miscoding affected 526 ARMS-managed

accounts.... By November 1998, the NG and

IS&T personnel had corrected the problem....”

prospectively

“...incoming e-mail to ARMS-managed

accounts with the first names beginning with

the letter ‘D’ had not been recorded by ARMS

since November 1998. It appears that this error

remained undetected until April 1999....”

Approximately 200 accounts within EOP were

affected.

“In the course of gathering these preliminary

facts concerning these configuration errors, we

were informed this week that e-mails on the

server of the Office of the Vice President (OVP)

have not been fully managed by ARMS.”

Exhibit 50

a

References are to the House Committee on Government Reform exhibits used at the March 2000

hearings; additional documents provided to the Committee are identified by an “Exxxx” scheme.

Bracketed items within the “Description of Event” column and the “Comments” column have been

added by us for readability and clarification.
Appendix III: Key Officials Involved in

Primary Mail2 and Letter D Events

Page 39 GAO-01-446 EOP E-mail System

Virginia Apuzzo, former Assistant to the President for Management and

Administration

Daniel A. “Tony” Barry, Computer Specialist, Office of Administration

(OA)/Information Systems and Technology Division (IS&T)

Mark Bartholomew, Web Development, OA/IS&T

Paulette Cichon, former Deputy Director, Information Management, OA

Dorothy Cleal, former Associate Director, OA/IS&T

Laura Crabtree, former Desktop Systems Branch Chief, OA/IS&T

Nellie Doering, Records Management, OA/IS&T

Kathleen Gallant, former Associate Director, OA/IS&T

Sheryl Hall, former Computer Specialist, OA/IS&T

Karl Heissner, Systems Integration and Development Branch Chief,

OA/IS&T

Joseph Kouba, Budget Analyst, Financial Management Division

Mark Lindsay, Assistant to the President for Management and

Administration, former General Counsel, OA, and former Director, OA

Michael Lyle, Director, OA, and former General Counsel, OA

Cheryl Mills, former Deputy Counsel to the President

Terrence Misich, Chief Warrant Officer 5, U.S. Army, on detail from White

House Communications Agency to OA/IS&T, Tape Restoration Project

Manager

Beth Nolan, Counsel to the President

Bruce Overton, Deputy General Counsel, OA

DeVere Patton, Contracting Officer’s Technical Representative (COTR) for

the Northrop Grumman Contract

Appendix III: Key Officials Involved in

Primary Mail2 and Letter D Events

Executive Office of

the President Federal

Staff, October 1996

Through June 1999:
Appendix III: Key Officials Involved in

Primary Mail2 and Letter D Events

Page 40 GAO-01-446 EOP E-mail System

John Podesta, Chief of Staff

Ada Posey, former Director, OA

Charles Ruff, former Counsel to the President

Michael Sullivan, Deputy Associate Director for General Services Division,

OA

Christina VanFossan, Director, Financial Management Division

James Wright, former COTR for the Northrop Grumman contract, and

Data Center Branch Chief, OA/IS&T

Sandra Golas, VAX System Administrator

Robert Haas, Lotus Notes E-mail System Administrator, Lotus Notes team

Steve Hawkins, former Project Manager

Betty Lambuth, CEXEC (a subcontractor to Northrop Grumman), former

Lotus Notes team manager

Joseph Lucente, Director of Contracts and Subcontracts

Yiman Salim, Lotus Notes Developer, Lotus Notes team

John Spriggs, Senior Engineer

Joseph Vasta, former Program Manager

Northrop Grumman

Contract Staff From

Contract Award in

October 1997 Through

June 1999:
Appendix IV: Time Line of Key Events in the

Mail2 and Letter D Malfunctions

Page 41 GAO-01-446 EOP E-mail System

Source: GAO compilation of documents provided by EOP to the House Committee on Government

Reform.

Appendix IV: Time Line of Key Events in the

Mail2 and Letter D Malfunctions
Appendix V: Itemization of Budgeted Cost for

the Tape Restoration Project

Page 42 GAO-01-446 EOP E-mail System

Cost element Description Estimated cost

ECS contract: Restoration of e-mail messages not recorded in ARMS

Phase I Task 1: Develop a detailed project plan $ 836,600

Task 2: Define, configure, acquire, and install required equipment 862,300

Task 3: Develop a quality assurance and quality control program for the project 217,600

Task 4: Develop, test, and refine tape recovery software 742,600

Task 5: Test and validate recovery plan and software 128,400

Phase II Task 6: Receive and inventory all tapes 118,000

Task 7: Make two copies of each tape 364,000

Task 8: Place original tapes in secure custody 31,000

Task 9: Scan duplicated tapes into message recovery system 337,000

Phase III Task 10: Process e-mail into searchable database 92,400

Task 11: Remove duplicate e-mail messages and prepare for transfer to ARMS 93,000

Task 12: Place duplicate e-mail messages into a searchable database 105,400

Task 13: Process classified tapes as above 208,000

Task 14: Deliver final products 80,000

General and administrative expenses and other direct costs 909,529

 ECS total: $5,125,829

Vistronix contract: Independent validation and verification of tape restoration

Phase I Tasks 1 through 5 (see above) $ 465,483

Phase II Tasks 6 through 9 (see above) 1,078,233

Phase III Tasks 10 through 14 (see above) 450,103

 Vistronix total: $1,993,819

Other costs: Other restoration costs

AAC Associates contract Advice on recovery of OVP e-mails $900,000

Vendor not determined Forensic analysis and recovery of broken or unreadable tapes 2,092,400

Vendor not specified Additional hardware, software licenses, and 12,000 tapes for two sets of tape copies 498,480

Not applicable Project overrun contingency 1,098,891

Other costs totals $4,589,771

Total $11,709,419

Source: Tape restoration project cost estimate report prepared by Vistronix, Incorporated, for the

EOP in July 2000.

Appendix V: Itemization of Budgeted Cost for

the Tape Restoration Project
Appendix VI: Comments From the Office of

the Former Vice President

Page 43 GAO-01-446 EOP E-mail System

Appendix VI: Comments From the Office of

the Former Vice President

Note: GAO comments

supplementing those in

the report text appear at

the end of this appendix.
Appendix VI: Comments From the Office of

the Former Vice President

Page 44 GAO-01-446 EOP E-mail System

Now on p. 2.

See comment 3.

Now on p. 1.

See comment 1.

Now on p. 2.

See comment 4.

Now on p. 2.

See comment 5.

Now on p. 1.

See comment 2.

Now on p. 2.

See comment 6.
Appendix VI: Comments From the Office of

the Former Vice President

Page 45 GAO-01-446 EOP E-mail System

Now on p. 10.

See comment 16.

Now on p. 7.

See comment 13.

Now on p. 5.

See comment 11.

Now on p. 4.

See comment 10.

See comment 9.

Now on pp. 2 to 3.

See comment 8.

Now on p. 10.

See comment 15.

Now on p. 2.

See comment 7.

Now on p. 9.

See comment 14.

Now on pp. 5 to 6.

See comment 12.
Appendix VI: Comments From the Office of

the Former Vice President

Page 46 GAO-01-446 EOP E-mail System

Now on p. 16.

See comment 25.

Now on p. 14.

See comment 24.

Now on p. 11.

See comment 28.

Now on p. 11.

See comment 22.

Now on p. 10.

See comment 21.

Now on p. 10.

See comment 20.

Now on p. 10.

See comment 19.

Now on p. 10.

See comment 18.

Now on p. 10.

See comment 17.
Appendix VI: Comments From the Office of

the Former Vice President

Page 47 GAO-01-446 EOP E-mail System

Now on p. 16.

See comment 26.
Appendix VI: Comments From the Office of

the Former Vice President

Page 48 GAO-01-446 EOP E-mail System

The following are GAO’s comments on the office of the former Vice

President’s letter, dated April 11, 2001.

1. We revised the report to include the word “certain,” but did not

incorporate the remaining text as suggested by the Office of the former

Vice President. The “certain” records that the report states are

“deemed official government records” are those incoming e-mail

messages not captured by the Automated Records Management

System (ARMS) due to the malfunctions. These records consisted of

both presidential/vice presidential and federal records.

2. We revised the report to reflect the technical clarification provided.

3. We submitted written questions to the Office of the Vice President

(OVP), through the Executive Office of the President (EOP), on

September 14, 2000, in which we asked if, and how, the OVP

implemented the portion of the 1997 EOP policy which stated that staff

“[m]aintain Presidential Records in organized files.” In its November 14

response, the OVP stated that OVP staff implemented the requirements

of the statute by maintaining presidential records “in organized files.”

In follow-up, we submitted another set of written questions to the OVP,

through the EOP, on November 30. At this time, our request was to

“[p]lease provide a list of all OVP staff who maintain the organized files

of hard copy OVP records as mentioned in the EOP response. We

would like to physically observe the files of randomly selected OVP

staff and would like to arrange for this observation during the week of

December 11, 2000.” Our purpose in reviewing the files of selected

staff was to respond to the House Committee on Government Reform’s

request that we determine whether OVP implemented adequate

practices in accordance with applicable criteria for management of email records. The intent of our request was to confirm the existence of

e-mail records in such files and not to examine the text of the records

contained therein, as explained verbally to the EOP Office of

Administration General Counsel on numerous occasions during

December 2000 and January 2001. We were not provided a contact

within OVP with whom to discuss the request. The OVP did not

respond to our request in writing until 6:49 p.m. on January 19, 2001, at

which time OVP’s response was that the OVP did not maintain a

centralized filing system. In its response, OVP may have confused the

word “organized” in our request with “centralized.” Several hours after

we were provided OVP’s response, the administration had transitioned

GAO Comments
Appendix VI: Comments From the Office of

the Former Vice President

Page 49 GAO-01-446 EOP E-mail System

and the records were in the process of transfer to the National

Archives and Records Administration.

4. A key feature of an electronic recordkeeping system is that records be

retrievable. To facilitate retrieval of records, the recordkeeping system

must index records and contain fields that can be searched.

5. Best practices for management of records currently exist in the form

of National Archives and Records Administration guidance contained

in title 36 of the Code of Federal Regulations (CFR). Even though the

majority of this guidance is prescriptive only for federal records, it

does not explicitly preclude application of these practices in the

management of presidential/vice presidential records. Although the

evolution of ARMS continues, the stability of generally accepted

records management best practices establishes a control foundation

for any system, whether manual or automated.

6. We revised the report to identify “another EOP component” as EOP’s

Office of Administration.

7. We revised the report to reflect the technical clarification provided.

8. We revised the report to reflect the technical clarification provided.

9. The report was revised to reflect that external authorities other than

those of law enforcement scrutinized the e-mail malfunctions.

10. We revised the report to incorporate the technical clarification

provided.

11. Title 36 CFR, subpart 1234.24, Standards for managing electronic mail

records, promulgates this standard (36 CFR 1234.24(c)).

12. The report provides a description of common federal standards for

management and control of federal programs and systems. We did not

imply that the application of such controls alters in any way the

determination of whether a record is presidential/vice presidential or

federal. The inclusion of this discussion was to emphasize that federal

programs, such as records management programs, and systems, such

as ARMS and the e-mail system, are subject to application of certain

management controls. These controls ensure the effectiveness of

programs as well as ensure that adequate integrity, confidentiality, and

availability of electronic information resources are maintained. Both
Appendix VI: Comments From the Office of

the Former Vice President

Page 50 GAO-01-446 EOP E-mail System

the Presidential Records Act (PRA) and the Federal Records Act

(FRA) require the implementation of management controls.

13. The new information provided in the Office of the former Vice

President’s comments on the draft has been incorporated into the

report.

14. We revised the report to reflect the technical clarification provided.

15. Under the PRA, the Vice President (and thus OVP), was responsible for

management of the Vice President’s records. From the beginning of the

administration, it retained the primary duty to implement records

management controls, and thus, to either modify its records

management practice to accommodate the system transition or to

ensure that this responsibility was assigned to another entity.

16. We attribute this statement to EOP because it was taken from the

White House memo to us on January 19, 2001, which stated that it

provided EOP’s responses to our questions submitted on November 30,

2000.

17. The new information provided by the Office of the former Vice

President has been incorporated into the report.

18. The new information provided by the Office of the former Vice

President has been incorporated into the report.

19. See our response to comment 3 above.

20. In its January 19, 2001, response to our written questions, the EOP

stated that at some point between issuance of the May 1993 EOP

policy and 1998, OVP ceased requiring the retention of paper copies of

e-mail records. We were provided no evidence, or allowed to

physically confirm, that some printed e-mail records were retained.

Also, the EOP policy did not explicitly require that records be captured

by ARMS, as indicated by the comment.

21. We revised the report to reflect the technical clarification provided.

22. As discussed in our response to comment 15 above, the OVP had a

duty to ensure continual preservation of its vice presidential records,

including ensuring that this responsibility effectively transferred to the

Office of Administration.
Appendix VI: Comments From the Office of

the Former Vice President

Page 51 GAO-01-446 EOP E-mail System

23. We revised the report to reflect the technical clarification provided.

24. The characteristics described in the report are based on National

Archives and Records Administration standards as presented in title 36

CFR. Creation of records is the manual or automated process during

which a document is produced and is determined to qualify as a

record. Identification is a part of this process in which there is

recognition that a document is a record and a determination is made as

to the type of record. The characteristics described pertain to the

monitoring of both presidential/vice presidential and federal records.

25. We agree with the Office of the former Vice President’s assessment;

however, the only rationale that was provided to us, either orally or in

writing, was that the letters were “deserving of respect and deference.”

We revised the report to incorporate the additional rationale provided

in its comments to the draft report.

26. The Committee requested that we assess the adequacy of management

controls over EOP’s records management system, particularly as they

relate to the numerous e-mail system malfunctions experienced by

EOP since 1996. That assessment entailed the examination of

management policies, programs, and practices as they related to

effective records management controls and to the requirements of the

PRA and FRA. Our review was not directed at assessing the intent of

individuals responsible for preserving records and providing

responsive records to investigative bodies. As such, we can offer no

conclusions on this matter.

(310307)

The first copy of each GAO report is free. Additional copies of reports are

$2 each. A check or money order should be made out to the

Superintendent of Documents. VISA and MasterCard credit cards are also

accepted.

Orders for 100 or more copies to be mailed to a single address are

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

Orders by visiting:

Room 1100

700 4th St., NW (corner of 4th and G Sts. NW)

Washington, DC 20013

Orders by phone:

(202) 512-6000

fax: (202) 512-6061

TDD (202) 512-2537

Each day, GAO issues a list of newly available reports and testimony. To

receive facsimile copies of the daily list or any list from the past 30 days,

please call (202) 512-6000 using a touchtone phone. A recorded menu will

provide information on how to obtain these lists.

Orders by Internet

For information on how to access GAO reports on the Internet, send an email message with “info” in the body to:

Info@www.gao.gov

or visit GAO’s World Wide Web home page at:

http://www.gao.gov

Contact one:

• Web site: http://www.gao.gov/fraudnet/fraudnet.htm

• E-mail: fraudnet@gao.gov

• 1-800-424-5454 (automated answering system)

Ordering Information

To Report Fraud,

Waste, and Abuse in

Federal Programs