GAO

United States General Accounting Office

Report to the Chairman of the 

Subcommittee on Defense, Committee on 

Appropriations, House of 

Representatives

October 1999 DEFENSE 

COMPUTERS

DOD Y2K Functional 

End-to-End Testing 

Progress and Test 

Event Management

GAO/AIMD-00-12

Page 1 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

Contents

Letter 3

Appendixes Appendix I: Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event Management 14

Appendix II: Objectives, Scope, and Methodology 47

Appendix III: Comments From the Department of Defense 49

Appendix IV: GAO Contact and Staff Acknowledgements 53

Tables Table 1: Summary of Recommended End-to-End Test Management 

Processes 6

Table 2: Summary of the Four Functional Area Test Approaches 7

Table 3: Summary of Test Events Satisfying GAO Key Processes 9

Abbreviations

DISA Defense Information Systems Agency

DOD Department of Defense

JUSE Joint User Switch Exercise

OMB Office of Management and Budget

OSD Office of the Secretary of Defense

PSA Principal Staff Assistant

Y2K Year 2000
Page 2 GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Page 3 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

United States General Accounting Office

Washington, D.C. 20548

Accounting and Information

Management Division

B-283564 Leter

October 18, 1999

The Honorable Jerry Lewis

Chairman, Subcommittee on Defense

Committee on Appropriations

House of Representatives

Dear Mr. Chairman:

Complete and thorough Year 2000 (Y2K) testing is essential to provide 

reasonable assurance that new or modified systems process dates correctly 

and will not jeopardize an organization’s ability to perform core business 

operations after the millennium. This is especially true for the Department 

of Defense (DOD) which relies on a complex and broad array of 

interconnected computer systems—including weapons, command and 

control, satellite, inventory management, transportation management, 

health, financial, personnel and payment systems—to carry out its core 

business functions and military operations. 

At your request, we initiated a review of the effectiveness of DOD’s efforts 

to perform Year 2000-related end-to-end tests for its major business 

functions, including Health Affairs, Communications, Personnel, and 

Logistics. Together, these functional areas are performing thousands of 

end-to-end tests to ensure that key business processes and systems can 

continue operating into the year 2000. Specifically, for each functional area, 

we analyzed reported information on the status and progress of all test 

events. We also selected and reviewed a critical test event in each 

functional area to determine whether it was planned and managed in 

accordance with our Year 2000 testing guide.1

 On September 14, 1999, we 

briefed you on the results of our review. This report provides a summary of 

our briefing and a recommendation to Defense for strengthening oversight 

of end-to-end testing for the Communications functional area. Subsequent 

to our briefing, logistics officials submitted additional information on the 

inclusion of installation telecommunications providers in related test 

events. We have clarified the briefing slides to reflect this. These 

clarifications, however, do not affect our overall conclusions and 

recommendation. The briefing slides are presented in appendix I, and our 

1

Year 2000 Computing Crisis: A Testing Guide (GAO/AIMD-10.1.21, November 1998).
B-283564

Page 4 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

objectives, scope, and methodology are in appendix II. The Office of the 

Assistant Secretary of Defense provided written comments on a draft of 

this report. These comments are discussed at the end of this report and 

reprinted in appendix III. We performed our audit work from March 

through September 1999 in accordance with generally accepted 

government auditing standards.

Results in Brief Because Year 2000 conversions often involve numerous large 

interconnecting systems with many external interfaces and extensive 

supporting technology infrastructures, Year 2000 testing should be 

approached in a structured and disciplined fashion. Our Year 2000 guidance 

recommends that in planning and managing end-to-end tests, agencies 

define test boundaries, secure the commitment of data exchange partners, 

prepare test procedures and data, define exit criteria,2

 and document test 

results, among other steps. Each of the individual test events we attended 

and reviewed within the four functional areas generally satisfied the key 

processes that our Year 2000 test guide defines as necessary to effectively 

plan, conduct, and report on end-to-end testing.3

 Moreover, while the 

events’ respective approaches to implementing the key processes varied, 

these differences were appropriately based on consideration of the event’s 

scope and complexity. 

In addition, overall end-to-end test efforts within three of the four 

functional areas were reported to be largely on schedule and expected to 

be completed by October 1999. However, at the time we briefed the 

Communications functional area on the results of our review, it could not 

provide complete progress information. While information was 

subsequently provided by Communications, it showed that the functional 

area had not yet developed plans to test 31 mission-critical systems. We are 

making a recommendation to Defense to ensure that these systems are 

tested or that there is adequate justification for their exclusion from end-toend test events. While Defense only partially concurred with this 

recommendation, it provided information showing the status of the 

systems in question. We did not verify this information.

2

Exit criteria are test conditions or requirements for successfully completing testing.

3

Our observations are limited to the specific events we witnessed, and we cannot draw 

conclusions regarding end-to-end testing from an overall functional area perspective.
B-283564

Page 5 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

Background In August 1998, the Deputy Secretary of Defense recognized the need to 

ensure that various key lines of business or functional areas within the 

department could continue to operate effectively at and after the turn of 

the century. Therefore, the Deputy Secretary directed Office of the 

Secretary of Defense focal points, known as Principal Staff Assistants 

(PSAs), to verify that all functions would be unaffected by Year 2000 issues. 

In doing so, the PSAs were to (1) document mission-critical functions and 

systems supporting those functions, (2) coordinate, facilitate, and monitor 

Year 2000 end-to-end test and evaluation activities of services, agencies, 

and commands, and (3) in some cases, conduct Y2K end-to-end functional 

testing.

The purpose of end-to-end testing is to verify that a defined set of 

interrelated systems, which collectively support an organizational core 

business area or function, interoperate as intended in an operational 

environment (either actual4

 or simulated). These interrelated systems 

include not only those owned and managed by the organization, but also 

the external systems with which they interface or that otherwise support 

the core business area or function.

The boundaries for end-to-end tests can vary depending on a given business 

function’s system dependencies and criticality to the organizational 

mission. Therefore, in managing end-to-end test activities, it is important to 

analyze the interrelationships among core business functions and their 

supporting systems, and the mission impact and risk of date-induced 

systems failures and to use these analyses to define test boundaries. It is 

also important to work early and continually with functional partners to 

ensure that related end-to-end test activities are effectively coordinated 

and integrated. As highlighted in table 1, our Year 2000 test guide, which 

has been adopted by the Office of Management and Budget (OMB), 

recommends that federal agencies take the following actions in planning 

and managing end-to-end tests.

4

Risks of testing in the production environment must be thoroughly analyzed and 

precautions taken to preclude damage to systems and data.
B-283564

Page 6 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

Table 1: Summary of Recommended End-to-End Test Management Processes

Define the system boundaries of the end-to-end 

test(s)

Agencies should define boundaries for the end-to-end test based on an assessment 

of their mission-critical business functions, inter- and intraorganization system 

dependencies, as well as the probabilities and impacts of any of these systems 

suffering a date-related failure.

Secure the commitment of data exchange 

partners

Because end-to-end testing addresses business areas or functions that involve 

multiple internal and external organizations, participation by all key data exchange 

partners should be solicited and obtained.

Establish an interorganizational test team A team composed of representatives from each of the organizations participating in 

the test should be formed to manage the planning, execution, and reporting of the 

test.

Confirm Year 2000 compliance of 

telecommunications infrastructure

In order to execute end-to-end testing and ensure that all systems in the chain of 

support to core business areas function as intended, agencies should ensure that the 

telecommunications infrastructure that interconnects the systems is compliant and 

ready for testing. 

Schedule and plan end-to-end test(s) A plan should be developed specifying key tasks and requirements for test planning, 

execution, and validation as well 25 milestones and resources associated with 

performing these tasks.

Prepare end-to-end procedures and data Interorganizational test procedures and data, including steps, cases, and input 

conditions that verify the correct handling of critical dates, should be prepared and 

approved by team representatives.

Define end-to-end test exit criteria The conditions or requirements for successfully completing end-to-end testing need to 

be established.

Execute end-to-end test(s) Tests should be executed in accordance with established plans and procedures.

Document test results Test results should be documented so that the data can be used to validate that test 

exit criteria had been met and to assess and correct problems discovered during the 

testing.

Correct Year 2000 defects On the basis of interoganization specified criteria, such as defect severity and test 

exist criteria, defects identified during the test should be prioritized and corrected.

Ensure that end-to-end test exit criteria are met Test results should be compared to test exit criteria to ensure that specified conditions 

are met.
B-283564

Page 7 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

The table below explains how the four functional areas included in our 

review approached their end-to-end tests.

Table 2: Summary of the Four Functional Area Test Approaches

The test events we selected from each area to review ranged from a simple 

test involving two information systems located within one organization to 

an intricate test of DOD’s voice and data telecommunications networks 

involving several commands and multiple systems. Specifically, the Health 

Affairs test event we reviewed assessed the ability of two interfacing 

systems to issue and process blood requests after the calendar year 

rollover. The Communications test event that we observed was a portion of 

a larger test and assessed whether voice communications could be sent 

from Fort Monmouth, New Jersey, to St. Louis, Missouri, using DOD 

telecommunications networks and equipment and whether messages could 

be exchanged using the Defense Messaging System from the Strategic 

Command to the Atlantic Command. The Personnel test event assessed the 

Army’s ability to create active duty units for deployment from the Army 

Reserve and Army National Guard. Finally, the Logistics event focused on 

intercomponent testing—between the Army, Air Force, Navy, Marine 

Corps, and Defense Logistics Agency—and was designed to verify the Year 

2000 readiness of 17 of the 53 total logistics requisition and receipt 

processes.

Function Description of decomposition

Health Affairs Health Affairs divided its function into three business processes: patient care, patient administration, and 

medical logistics. Health Affairs then broke down each process into several sub-processes, termed “threads.”

Communications Because Communications cross cuts all functional/operational areas, Communications is testing based on 

system user. Therefore, Communications divided its function into 263 mission-critical systems. Various military 

services, Defense agencies, and commanders-in-chief own these systems.

Personnel Personnel divided its function into six areas: Army, Navy, Air Force, Marine Corps, Civilian, and DEERS/RAPIDS 

(personnel systems). Personnel then broke down each area into sub-processes, termed “threads.” Personnel is 

not conducting its own end-to-end tests. Instead, Personnel is participating in and observing service-level 

testing. 

Logistics Logistics divided its function into four business processes: requisition, receipt, shipment, and inventory control 

and asset status. Logistics then broke down each process into several sub-processes, termed “threads.” 

Logistics tested these four processes in two phases: intracomponent (within each military service or Defense 

agency) and intercomponent (joint testing with military services and Defense agencies).
B-283564

Page 8 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

End-to-End Tests 

Reported to Be on 

Schedule

Available information for the respective areas indicates that, as of August 

1999, end-to-end tests were largely on schedule and expected to be 

completed by October 1999. In particular, 

• Health Affairs, which had three primary business processes, completed 

testing for two—patient care and patient administration business 

processes—and was on schedule to complete tests for the third—

medical logistics—by the end of September 1999. 

• Personnel tests for the Army, Air Force, and Civilian areas had been 

completed, while the Navy tests were scheduled to be done October 17, 

1999. The Marine Corps was behind schedule on one test. However, it 

completed the test by September 9, 1999. 

• Logistics intra- and intercomponent tests, which involve four primary 

business processes—requisition, shipment, receipt, and inventory 

control and asset status—had been completed for intercomponent 

transactions. Tests were scheduled to be done by the end of August 1999 

and, according to Logistics officials, were completed on schedule.

• When we briefed the Communications functional area on the results of 

our review in July 1999, it was unable to provide progress information 

on all of its 263 mission-critical systems. Subsequently, Communications 

reported that 77 mission-critical systems had completed testing and 155 

systems did not require testing.5

 The functional area also reported that 

the remaining 31 mission-critical systems6 did not yet have plans for 

testing and were considered to be behind schedule.

Selected End-to-End 

Test Events Were 

Managed According to 

GAO Guidance

We selected one test event from each functional area, determined whether 

the key processes outlined in our Year 2000 testing guide were followed 

and found that DOD had completed the majority of the processes called for 

in the guide. For example, for the four test events reviewed, DOD had 

defined test boundaries, defined exit criteria that would be used to 

determine when a test was successfully completed, and described how the 

test results would be documented. While the event’s respective approaches 

to implementing the key processes varied, these differences were based on 

5

The Communications function considers systems that do not require end-to-end testing to 

be developmental systems, those that do not process dates, and stand-alone systems.

6

According to Communications officials, some of these systems are satellite and control 

systems, which may require waivers.
B-283564

Page 9 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

the consideration of the event’s scope and complexity and inherent 

business risk. Our test guidance permits such differences when justified on 

the basis of business value and risk.

Table 3 summarizes the results of our review. As the table notes, of the 

possible 44 key processes spanning the 4 test events, 34 were fully satisfied 

while another 2 were partially satisfied. For the remaining 8 key processes, 

4 were still in progress, and 4 processes concerning correcting defects 

found were “not applicable” because initial testing results had not yet 

disclosed Year 2000 defects. However, some of the test results that were 

obtained during our review were still being analyzed by DOD. 

Table 3: Summary of Test Events Satisfying GAO Key Processes

Note: Due to differences in scope and complexity of the test events, the results of individual functions 

are not comparable.

In all cases where we determined that the test events’ key processes called 

for in our guide had only been partially satisfied, the PSAs and test 

managers agreed to address our concerns and initiate corrective actions. 

For example:

• While Health Affairs prepared procedures for its test event, these 

procedures were not sufficiently detailed and did not define each step to 

be executed or precisely define input data. As a result, it was necessary 

for system operators to augment the test procedures during the test’s 

execution. While this approach was satisfactorily carried out because 

the relative simplicity of the test event permitted face-to-face 

coordination and synchronization of the procedures, it was 

unnecessarily risky and could have been easily avoided by ensuring that 

test procedures were complete. Health Affairs officials agreed that more 

detailed procedures should have been established, and they committed 

to ensuring that other Health Affairs test events have them. 

Selected functional area test event Fully satisfied Partially satisfied In progress N/A Total

Health Affairs 8 1 1 1 11

Communications 9 0 1 1 11

Personnel 9 0 1 1 11

Logistics 8 1 1 1 11

Total 34 2 4 4 44
B-283564

Page 10 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

• Although the Logistics function is reliant on telecommunications 

providers such as military installations and the Defense Information 

Systems Agency (DISA), at the time of the test event we observed, 

documentation offering assurances that installations’ 

telecommunications infrastructures were Y2K compliant was not 

provided by Logistics functional managers. Our test guide states that, in 

order to ensure that all systems in the chain of support function as 

intended, the telecommunications infrastructure that interconnects the 

systems must be compliant and ready for testing. Subsequent to our 

review, Logistics officials provided information showing that 

installations’ telecommunications infrastructures had been included in 

installation test events. Logistics officials agreed, however, that they had 

not yet confirmed the Y2K compliance of the infrastructures, and 

reported that they have subsequently initiated steps to do so.

Conclusions Given that virtually all Defense business functions and military operations 

rely heavily on technology, it is vital that Year 2000 end-to-end testing 

efforts be effectively planned and executed. All four of the individual test 

events that we reviewed were well-managed because each either satisfied 

or had steps underway or planned to address all relevant end-to-end 

management key processes specified in our test guide. Moreover, 

differences between the functional areas’ approaches to implementing 

these key processes were generally commensurate with the events’ scope 

and complexity. Finally, reported functional area status information 

indicates that end-to-end tests are generally progressing on schedule. 

However, DOD does not yet have assurance that all of its communications 

systems will be Year 2000 compliant and, as such, should ensure that all 

mission-critical communications systems are tested.

Recommendation We recommend that the Secretary of Defense direct the Senior Civilian 

Official of the Office of the Assistant Secretary of Defense for Command, 

Control, Communications, and Intelligence to report to the Deputy 

Secretary immediately on plans for end-to-end testing the 31 missioncritical communications systems, including milestones for executing tests 

and reporting test results, or to otherwise justify in writing to the Deputy 

Secretary why any of the systems will not be included in an end-to-end test 

event. 
B-283564

Page 11 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

Agency Comments and 

Our Evaluation

DOD concurred with our findings and partially concurred with our 

recommendation to report to the Deputy Secretary on the status and plans 

for Y2K testing of the 31 mission-critical communications systems 

disclosed in our report.

In partially concurring on the recommendation, DOD stated that during the 

July through August 1999 period of our review, testing data in the OSD Y2K 

database was still evolving, and as a result, test data were incomplete for 

many of the 31 systems. Since then, resolution has been reached on the 

testing status of the 31 communications systems. DOD reported and 

provided documentation to show that (1) Y2K testing for 14 of the 31 

systems has been completed, (2) 9 systems do not process dates and are 

exempt from end-to-end test requirements, (3) 4 systems are trusted 

systems, which cannot be tested in a Y2K environment due to safety, 

security, or operational necessity reasons, (4) 2 systems are developmental 

systems that will not be deployed before the millennium rollover, (5) 1 

system has been reclassified as a nonmission-critical system and does not 

require additional testing, and (6) 1 system is scheduled to complete testing 

by October 15, 1999. We have not verified the status information provided 

by DOD. 

We are sending copies of this report to Representative John P. Murtha, 

Ranking Minority Member, Subcommittee on Defense, House 

Appropriations Committee; Senator John Warner, Chairman, and Senator 

Carl Levin, Ranking Minority Member, Senate Committee on Armed 

Services; Senator Ted Stevens, Chairman, and Senator Daniel Inouye, 

Ranking Minority Member, Subcommittee on Defense, Senate Committee 

on Appropriations; and Representative Floyd Spence, Chairman, and Ike 

Skelton, Ranking Minority Member, House Committee on Armed Services.

We are also sending copies to the Honorable John Koskinen, Chair of the 

President’s Year 2000 Conversion Council; the Honorable William Cohen, 

Secretary of Defense; the Honorable John Hamre, Deputy Secretary of 

Defense; General Henry Shelton, Chairman of the Joint Chiefs of Staff; 

Arthur Money, Senior Civilian Official of the Office of the Assistant 

Secretary of Defense for Command, Control, Communications, and 

Intelligence; and the Honorable Jacob Lew, Director of the Office of 

Management and Budget. Copies will also be made available to others upon 

request.
B-283564

Page 12 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

Should you or your staff have any questions concerning this report, please 

contact me at (202) 512-6240. I can also be reached by e-mail at 

brockj.aimd@gao.gov. Other points of contact and key contributors to this 

report are listed in appendix IV.

Sincerely yours,

Jack L. Brock, Jr.

Director, Governmentwide and Defense

Information Systems
B-283564

Page 13 GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Page 14 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

Appendix I

Briefing on DOD Y2K Functional End-to-End Apendxies

Testing: Progress and Test Event Management ApendixI

DOD Y2K Functional End-to-End

Testing: Progress and Test Event

Management

House Appropriations Committee

September 14, 1999
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 15 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

2

Briefing Overview

• Introduction

• Objectives

• Scope and Methodology

• Background

• Objective 1: Status and Progress Information Summary

• Objective 2: Results of Specific Test Events Reviewed

• Findings

• Conclusions

• Recommendation
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 16 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

3

Introduction

• In August 1998, the Deputy Secretary of Defense directed

the Principal Staff Assistants (PSAs), who are responsible

for Defense’s functional areas, to coordinate, facilitate and

monitor a series of Year 2000 functional evaluations (Y2K

functional testing) to ensure functions will operate properly

in a Y2K environment.

• Y2K functional testing is one of three Defense Y2K endto-end test and evaluation efforts. Y2K end-to-end testing

is testing performed to verify that systems collectively

supporting an organizational core business function or

operation interoperate as intended in a Y2K environment.

• The PSAs core business processes are referred to as

“threads.”
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 17 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

4

GAO Objectives

• At the request of the Chairman, House Appropriations

Committee, Defense Subcommittee, we initiated a

review of the effectiveness of DOD Y2K functional endto-end test management. The objectives of the review

are to:

• Assess status/progress on all test events within four

functional areas: Health Affairs, Communications,

Personnel, and Logistics

• Review the management effectiveness of a critical

test event for each of the four functional areas
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 18 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

5

Scope and Methodology

• The test events we reviewed were:

• Health Affairs

Patient Care/Issuance and Processing of Blood

Requests

• Communications

Joint User Switch Exercise (JUSE-99-Y2K)

• Personnel

Army Personnel/Mobilization/Reserve Unit

• Logistics

Joint tests of requisition and receipt processes
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 19 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

6

Scope and Methodology

• For all test events, we compared reported status

information to test plan milestones to identify variances.

• For the selected test events, we:

• observed a part of the test event for each functional

area and

• compared event management activities to the key

processes for end-to-end testing as specified in our

Year 2000 Test Guide.

• The test events we reviewed were selected based on the

PSA’s designation that the events were of key importance

to the business function.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 20 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

7

Scope and Methodology

• The test events we reviewed assessed calendar year

rollover only.

• We briefed the respective PSAs and test event

managers on the results of our reviews following the

conclusion of each event.

• Health Affairs, June 15, 1999

• Communications, July 12, 1999

• Personnel, July 15, 1999

• Logistics, August 17, 1999
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 21 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

8

Scope and Methodology

• In commenting on the briefings, PSAs and test managers agreed

with the results. This briefing consolidates the results of those

individual briefings, and provides updated status information on

each functional area’s testing progress. We also clarified the

results of our analysis of the Logistics test event based on

additional information provided by Logistics officials after our

September 14, 1999 briefing.

• Due to differences in scope and complexity, the results of our

review of the test events are not comparable across functions.

• We performed our audit work in accordance with generally

accepted government auditing standards between March and

September 1999.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 22 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

9

Background

• DOD requires PSAs to verify that their respective critical

business processes can function properly in a Y2K

environment.

• PSAs are to:

• document mission-critical functions and systems

supporting those functions,

• coordinate, facilitate, and monitor Y2K end-to-end test

and evaluation activities of services, agencies, and

commands, and

• in some cases, conduct Y2K end-to-end functional

testing.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 23 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

10

Background

• The scope and complexity of the test events varied in

terms of, for example, the numbers of systems tested and

locations involved.

• Health Affairs test event involved a two system cluster

in a simulated Y2K environment. In contrast, the

Logistics test event included 6 operational systems and

multiple geographical dispersed installations.

• Personnel test event involved only Army systems.

Communications, Logistics and Health Affairs test

events involved more than one Defense component.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 24 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

11

Background

• Army Personnel and Logistics chose different methods for

testing threads that spanned more than one day.

• Army Personnel divided its threads into what are

termed “pairs” and tested transactions across each

“pair” of systems using the critical rollover date.

• Logistics used a 5-day test cycle that commences on

the day before rollover (i.e., 9/30/1999, 12/31/1999, and

2/28/2000). This strategy verifies the mission-critical

thread across the date rollover, rather than verify all

systems ability to correctly perform their respective

function across the date rollover.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 25 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

12

Objective 1: Status and

Progress Information Summary

• Criteria: Management should collect timely, reliable test

progress and status information in order to ensure that

testing is progressing according to plans and that

problems are resolved and testing requirements are met.

• Finding: Health Affairs, Personnel, and Logistics are

collecting the progress and status information.

• Finding: At the time of our July 1999 briefing to the PSA,

Communications was unable to provide us with complete

information on functional test event’s progress for all

mission-critical systems it had identified. Subsequently,

additional information was provided that shows the status

of all systems; however, Communications reported that

tests have not yet been planned for 31 of the systems.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 26 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

13

Objective 1: Status and

Progress Information Summary

(continued)

• Finding: With the exception of the tests that need to be

planned for the 31 Communications systems, the

available test progress information shows that functional

areas end-to-end test events are on schedule.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 27 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

14

Objective 1: Status and

Progress Information

Health Affairs

Test Type Reported Status of Testing Efforts

• Patient Care 26 out of 26 threads have been tested.

Testing was completed on schedule.

• Patient

Administration

30 out of 30 threads have been tested.

Testing was completed on schedule.

• Medical

Logistics

26 out of 55 threads have been tested.

Testing commenced on July 2, 1999 and is on schedule

to be completed by the end of September 1999.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 28 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

15

Objective 1: Status and

Progress Information (continued)

Communication

sTest Type Reported Status of Testing Events

• Army Of 68 systems:

19 have been tested, 4 have not been planned and are late, and 45 systems

do not require tests.1

• Air Force Of 32 systems:

10 have been tested, 14 have not been planned and are late, and 8 systems

do not require tests.1

• Navy Of 123 systems:

30 have been tested, 4 have not been planned and are late, and 89 systems

do not require tests.1

• Marine

Corps

Of 6 systems:

1 has been tested, 1 has not been planned and is late, and 4 systems do not

require tests.1

• CINCs Of 15 systems:

5 have been tested, and 8 have not been planned and are late, and 2

systems do not require tests.1

• DISA Of 19 systems:

12 have been tested and 7 systems do not require tests.1

1Communications relied on the Y2K Office database to identify systems that do not require testing. Systems that do not require tests are: 1) developmental systems, 2) do not process

dates, or 3) are stand alone systems.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 29 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

16

Objective 1: Status and

Progress Information (continued)

Test Type Reported Status of Testing Efforts

• Army 19 out of 19 threads have been tested.

Testing was completed on schedule.

• Navy Testing is in progress, 0 out of 18 threads

are complete. The testing was started ahead

of schedule and is planned to be completed

by October 17, 1999.

• Marine

Corps

7 out of 7 threads have been tested.

The testing began on time; however, 1 thread

was not completed until September 9, 1999,

instead of August 20, 1999.

• Air Force 10 out of 10 threads have been tested.

Testing was completed on schedule.

• Civilian 2 out of 2 threads have been tested.

Testing was completed on schedule.

• DEERS/

RAPIDS2

4 out of 4 threads have been tested.

Testing was completed on schedule.

Personnel

2Defense Enrollment Eligibility Reporting Systems (DEERS), Real Time Automated Personnel ID Card System (RAPIDS)
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 30 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

17

Objective 1: Status and

Progress Information (continued)

Test Type Reported Status of Testing Efforts

• Intra-component All components have completed testing.

• Inter-component

(Requisition)

41 of 41 threads have been tested.

Testing was completed on schedule.

• Inter-component

(Shipment)

4 of 4 threads have been tested.

Testing was completed on schedule.

• Inter-component

(Receipt)

4 of 4 threads have been tested.

Testing was completed on schedule.

• Inter-component

(Inventory Control &

Asset Status)

4 of 4 threads have been tested.

Testing was completed on schedule.

Logistics
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 31 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

18

Satisfied

In Progress

Partially

Satisfied

Not 

Applicable

Objective 2: Results of Specific

Test Events Reviewed

3The functional areas were abbreviated: Health Affairs (HA), Communications (Comm), and Personnel (Pers)

GAO Test Criteria HA3 Comm3 Pers3 Logistics

Define the system boundaries of the end-to-end

test(s)

Secure the commitment of data exchange partners

Establish an inter-organizational test team

Confirm Y2K compliance of telecommunications

Schedule and plan end-to-end test(s)

Prepare end-to-end procedures and data

Define end-to-end test exit criteria

Execute end-to-end test(s)

Document end-to-end test results

Correct Y2K defects

Ensure that end-to-end test exit criteria are met

Note: Due to differences in scope and complexity of the test events, the results of individual functions are not comparable.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 32 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

19

Finding: End-to-end test

boundaries defined

• Criteria: Organizations need to assess their mission-critical

business functions and dependencies and, on the basis of

this assessment, define boundaries for end-to-end tests.

• Finding: All of the functional areas defined test boundaries

for the test events we witnessed. For example,

Communications defined boundaries, including geographic

sites, deployed forces, satellites, and control centers.

Health Affairs identified three critical functions: Patient Care,

Patient Administration, and Medical Logistics. Patient Care

was broken down into mission-critical threads, one of which

was issuing and processing blood requests.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 33 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

20

Finding: Commitment of key

data exchange partners

secured

• Criteria: Because end-to-end testing addresses business

areas or functions that involve multiple organizations,

participation of all key business area data exchange

partners should be secured.

• Finding: All of the functional areas secured the

commitment of key data exchange partners. For example,

Communications data exchange partners from different

parts of the world agreed to send and receive data,

including Ft. Monmouth, St. Louis, Atlantic Command, and

Strategic Command for the test that we reviewed.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 34 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

21

Findings: Interorganizational

end-to-end teams established

• Criteria: A team composed of representatives from each

of the organizations participating in the test should be

formed to manage the planning, execution, and reporting

of the test.

• Finding: All functional areas established an interorganizational team that planned, executed, and reported

on the test event. For example, Health Affairs established

a team that included the TRICARE Management Activity

Office, IV&V contractors, and technical support staff from

Advanced Technology Integration Center.

• Finding: Logistics interorganizational team included

relevant data exchange partners, such as the military

services and Defense Logistics Agency.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 35 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

22

Findings: Y2K compliance of

telecommunications partially

ensured

• Criteria: In order to ensure that all systems in the chain of

support function as intended, the telecommunications

infrastructure that interconnects the systems must be

compliant and ready for testing.

• Finding: Health Affairs ensured that its

telecommunications infrastructure for the test (i.e., local

area network) was Y2K compliant.

• Finding: Communications, Personnel, and Logistics endto-end test events used the Unclassified (but Sensitive)

Internet Protocol Router Network (NIPRNET). Defense

Information Systems Agency reported that the NIPRNET

has been certified Y2K compliant and is fully

implemented.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 36 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

23

Findings: Y2K compliance of

telecommunications partially

ensured (continued)

• Finding: Communications and Personnel have verified

that their local telecommunications infrastructure are Y2K

compliant. Logistics had not verified that its local

telecommunications infrastructure is Y2K compliant, but

has since initiated steps to do so.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 37 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

24

Finding: End-to-end tests

planned and scheduled

• Criteria: A plan should be developed that defines key

tasks and requirements in preparing for, executing, and

documenting results of testing. Responsibility for

performing key tasks and quality assurance/Independent

Verification and Validation (IV&V) need to be established.

• Finding: All functional areas established plans that

identified key tasks, schedules, roles and responsibilities

for the participants in the test events. For example,

detailed plans and schedules were provided in the Health

Affairs Master Test Plan and Patient Care Event Plan,

which addressed the IV&V contractor’s roles and

responsibilities. Also, Personnel had an Army Master

Test Plan and Army System Test Plans that identified key

tasks, schedules, roles and responsibilities.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 38 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

25

Findings: End-to-end test

procedures prepared

• Criteria: Test procedures and data should include steps,

cases, and input conditions that verify the correct handling

of critical dates.

• Finding: Most functional areas prepared test procedures

and data. For example, Army Personnel prepared baseline

input test data and generated baseline output. Army

Personnel then aged the baseline test data inputs and

outputs to reflect a Y2K environment and used these to

verify correct handling of the calendar year rollover date.

• Finding: Health Affairs developed high-level test

procedures; however, it was necessary for operators to

augment the test procedures to complete the test. Health

Affairs officials agreed that more detailed procedures

should have been established.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 39 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

26

Finding: Test exit criteria

defined

• Criteria: The conditions or requirements for successfully

completing end-to-end testing need to be established.

• Finding: All functional areas defined test exit criteria for

the specific test events. For example, Communications

specified that the test exit criteria for successfully

completing voice communications were to hear a person

at the other end of the line with a specified clarity. It

specified that the test exit criteria for successfully

completing data communications were for all the message

recipients to receive the “test” message with all

attachments over a 72-hour period.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 40 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

27

Finding: End-to-end test plans

were followed

• Criteria: Tests should be executed in accordance with

established plans and procedures.

• Finding: All test events adhered to the established plans

and procedures. For instance, the Army Personnel test

event was executed in accordance with both the master

and system test plans. Similarly, the Logistics test event

was executed in accordance with the test plans and

procedures.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 41 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

28

Finding: Test results were

documented

• Criteria: End-to-end test results should be documented so

that the data can be used to validate that test exit criteria

have been met and to assess and correct problems

discovered during testing.

• Finding: All functional areas documented test results.

Both Health Affairs and Personnel documented results by

printing out computer screen information. Logistics and

Communications documented test results on a results

reporting form.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 42 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

29

Finding: No Y2K defects were

found during the test event

• Criteria: On the basis of interorganization specified

criteria, such as defect severity and test exit criteria,

defects identified during the test should be prioritized and

corrected.

• Finding: Initial results from the part of the test event that

we observed indicated that none of the functional areas

experienced Y2K defects. However, analysis of the test

results has not yet been completed by all of the functional

area data analysis centers.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 43 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

30

Finding: Effort to ensure

end-to-end test meets exit

criteria is ongoing

• Criteria: Test results should be compared to test exit

criteria to ensure that specified conditions are met.

• Finding: All functional areas are still reviewing test results

at the data collection centers. For example, Logistics’

Exercise Operation Center is collecting results reporting

forms and assessing the satisfaction of test exit criteria.

In addition, Communications-Electronics Command is

collecting test results for analysis and determination of

test exit criteria satisfaction.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 44 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

31

Conclusions

• Three of the four PSAs were able to provide us with

information needed to effectively track the progress and

status of their respective inventories of end-to-end test

events. Communications was not initially able to provide

this information on all its test events. While additional

status information was provided recently, this information

identified mission-critical systems for which end-to-end

testing had not yet been planned, thus leaving very little

time to ensure that they are included in an end-to-end test

event.

• The four test events that we reviewed were effectively

managed based on the satisfaction of most, if not all, of

the key end-to-end testing processes in our guidance.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 45 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

32

Conclusions

• The Logistics test event was somewhat limited by not

ensuring the Year 2000 compliance of the end-to-end

partners’ local telecommunications equipment connecting

the partners to the NIPRNET. Logistics has initiated

appropriate steps to do so.

• The test events’ respective approaches and the degree of

test planning, execution, and reporting on the four test

events varied. However, the variances were generally

commensurate with the unique characteristics of the

respective events. Such variability in test approaches is

consistent with our test guidance.
Appendix I

Briefing on DOD Y2K Functional End-to-End 

Testing: Progress and Test Event 

Management

Page 46 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

33

Recommendation

• We recommend that the Secretary of Defense direct the

Senior Civilian Official of the Office of the Assistant

Secretary of Defense for Command, Control

Communications, and Intelligence to report to the Deputy

Secretary immediately on plans for end-to-end testing the

31 systems, including milestones for executing tests and

reporting test results, or to otherwise justify in writing to

the Deputy Secretary why any of the systems will not be

included in an end-to-end test event.
Page 47 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

Appendix II

Objectives, Scope, and Methodology ApendixI

As requested by the House Committee on Appropriations, Subcommittee 

on Defense, our objectives were to (1) assess the status and progress of all 

test events within four functional areas—Health Affairs, Communications, 

Personnel, and Logistics—and (2) review the management effectiveness of 

a critical test event for each of the four functional areas. Together, these 

functional areas are performing thousands of end-to-end tests to ensure 

that key business processes and systems can continue operating into the 

year 2000.

To meet our first objective, we obtained status and progress information 

for the aforementioned functional areas and compared the reported status 

information to milestones contained in individual functional test plans1

 to 

identify variances. We discussed this information with DOD officials and 

personnel from the Office of the Assistant Secretary of Defense (Health 

Affairs), the Office of the Assistant Secretary of Defense for Command, 

Control, Communications, and Intelligence, the Office of the 

Undersecretary of Defense (Personnel and Readiness), and the Deputy 

Undersecretary of Defense (Logistics). Also, for each of the functional 

areas, where necessary, we obtained updated status and progress 

information on end-to-end test events.

To meet our second objective, we selected one specific test event for each 

functional area. The four selected test events were based on each PSA’s 

designation that the test event was of key importance in ensuring that the 

function could continue unaffected at and after the turn of the century. The 

selected test events, and the dates and locations we observed the events, 

were

• Health Affairs—Patient Care/Issuance and Processing of Blood 

Requests (May 18, 1999) at the Advanced Technology Integration Center 

in Falls Church, Virginia.

• Communications—Joint User Switch Exercise (JUSE-99-Y2K)

(June 10, 1999) at the Army Communications-Electronics Command, 

Fort Monmouth, New Jersey.

• Personnel—Army Personnel/Mobilization/Reserve Unit (June 17, 1999) 

at the Army Personnel Command, Alexandria, Virginia.

1

Updated plans included in our review were the December 15, 1998, plan for 

communications; the January 1999, plan for health affairs; the January 31, 1999, plan for 

logistics; and the January 28, 1999, plan for personnel.
Appendix II

Objectives, Scope, and Methodology

Page 48 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

• Logistics—Intercomponent test of requisition and receipt processes 

(June 24-25, 1999) at the Navy Fleet Material Support Office in 

Mechanicsburg, Pennsylvania. 

For the selected test events, we interviewed DOD officials and reviewed 

pertinent documentation for each event, including test event plans,2

procedures, conditions, exit criteria, results, reports, defects, correction 

action plans, and we observed the actual execution of the test event. We 

then compared the particulars of each event to our Year 2000 test guide’s 

end-to-end testing key processes, identified variances, and discussed with 

test officials the reasons for and impacts of any variances.

To supplement our documentation reviews and observations, we 

interviewed DOD officials, including those from the TRICARE Military 

Health Systems, the Army Communications-Electronics Command, the 

Office of the Deputy Chief of Staff for Personnel, and the Office of the 

Deputy Undersecretary of Defense (Logistics); test event coordinators; and 

test directors regarding additional clarifications after our visit. These 

officials addressed telecommunications infrastructure Year 2000 

compliance issues. They also provided additional documentation from the 

test event we witnessed (i.e., test results, quick look, and final reports). 

Due to the time criticality of the year 2000, as our reviews were completed 

on each of the functional areas, we provided briefings detailing our 

observations to each Defense PSA and test director as follows:

• Health Affairs—June 15, 1999,

• Communications—July 12, 1999,

• Personnel—July 15, 1999, and

• Logistics—August 17, 1999.

We performed our audit work primarily at DOD headquarters and at the 

test event locations described above. We requested and received comments 

on a draft of this report from DOD and incorporated those comments as 

appropriate. We performed our audit work from March through September 

1999 in accordance with generally accepted government auditing 

standards.

2

Test event plans included in our review were: Military Health Systems Patient Care 

Functional Readiness Assessment Test Plan, v1.2, April 1999; Communications Joint User 

Switch Exercise 99-Y2K Exercise Directive, April 1999; Army Personnel Command Test 

Plan, June 1999; Army System Test Plans, June 1999, for EDAS, SIDPERS-3, and PEPDUS; 

and Logistics Exercise Directive, May 1999.
Page 49 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

Appendix III

Comments From the Department of Defense ApendxiI
Appendix III

Comments From the Department of Defense

Page 50 GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Appendix III

Comments From the Department of Defense

Page 51 GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Appendix III

Comments From the Department of Defense

Page 52 GAO/AIMD-00-12 DOD Y2K End-to-End Testing
Page 53 GAO/AIMD-00-12 DOD Y2K End-to-End Testing

Appendix IV

GAO Contact and Staff Acknowledgements ApendixIV

GAO Contact Randolph C. Hite, (202) 512-6240

Acknowledgements In addition to those named above, Ronald B. Bageant, Cristina T. Chaplain, 

Katherine I. Chu, Richard B. Hung, Myong S. Kim, Madhav S. Panwar, and 

Alicia L. Sommers made key contributions to this report.

(511659) Letr
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the 

following address, accompanied by a check or money order made 

out to the Superintendent of Documents, when necessary, VISA and 

MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are 

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and 

testimony. To receive facsimile copies of the daily list or any list 

from the past 30 days, please call (202) 512-6000 using a touchtone 

phone. A recorded menu will provide information on how to obtain 

these lists.

For information on how to access GAO reports on the INTERNET, 

send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at: 

http://www.gao.gov

United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. GI00