GAO

United States General Accounting Office

Report to the Chairman of the 

Subcommittee on Defense, Committee on 

Appropriations, House of 

Representatives

November 1999 DEFENSE 

COMPUTERS

U.S. Space Command’s 

Management of Its 

Year 2000 Operational 

Testing

GAO/AIMD-00-30

Page 1 GAO/AIMD-00-30 Space Command Y2K Operational Testing

Contents

Letter 3

Appendixes Appendix I: Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL 14

Appendix II: Objectives, Scope, and Methodology 67

Appendix III: GAO Contact and Staff Acknowledgements 68

Tables Table 1: Summary of JCS Year 2000 Operational Evaluation

Criteria 7

Table 2: Summary of Space Command’s Satisfaction of JCS

Evaluation Criteria for the Space Control Evaluation 8

Abbreviations

DOD Department of Defense

JCS Joint Chiefs of Staff

OPEVAL operational evaluation

SPACECOM U.S. Space Command

Y2K Year 2000
Page 2 GAO/AIMD-00-30 Space Command Y2K Operational Testing
Page 3 GAO/AIMD-00-30 Space Command Y2K Operational Testing

United States General Accounting Office

Washington, D.C. 20548

Accounting and Information

Management Division

B-282546 Leter

November 15, 1999

The Honorable Jerry Lewis

Chairman, Subcommittee on Defense

Committee on Appropriations

House of Representatives

Dear Mr. Chairman:

Complete and thorough end-to-end testing is essential to provide 

reasonable assurance that new or modified systems used to collectively 

support a core business function or mission operation will not jeopardize 

an organization’s ability to deliver products and services as a result of the 

Year 2000 (Y2K) computing problem. This is especially true for the 

Department of Defense (DOD) because it relies on a complex and broad 

array of interconnected computer systems—including weapons, command 

and control, satellite, inventory management, transportation management, 

health, financial, personnel and payment systems—to carryout its military 

operations and supporting business functions.

At your request, we reviewed DOD’s management of various Year 2000-

related end-to-end testing activities. As part of our efforts, we assessed the 

U.S. Space Command’s management of its end-to-end test of space control 

systems essential to major theater war (one of 16 operational evaluations 

for the command) and determined what the results of this test show with 

respect to operational risks and readiness. 1

 We briefed Space Command 

officials on our findings on October 1, 1999, and made a recommendation 

to correct the management weaknesses that we found. Space Command 

immediately acted to address our recommendation. We then briefed your 

office on our findings and Space Command’s actions to address our 

recommendation on November 1, 1999. The purpose of this letter is to 

summarize our briefing to your office. The briefing slides that we presented 

to your office are in appendix I, and the objectives, scope, and 

methodology of our review are detailed in appendix II. Space Command 

provided oral comments on our briefing slides, and we have incorporated 

them as appropriate. We performed our audit work from March through 

1

DOD refers to its combatant commands’ end-to-end tests as operational evaluations.
B-282546

Page 4 GAO/AIMD-00-30 Space Command Y2K Operational Testing

October 1999 in accordance with generally accepted government auditing 

standards.

Results in Brief Year 2000 end-to-end testing is an essential component of an effective Year 

2000 testing program since Y2K-related problems can affect so many of the 

systems owned and operated by an entity as well as systems belonging to 

business partners and infrastructure providers. Moreover, to be effective, 

end-to-end testing should be approached in a structured and disciplined 

fashion. Both the Joint Chiefs of Staff (JCS) guidance to its combatant 

commands on managing Year 2000 operational evaluations,2 (the term JCS 

uses to refer to Year 2000 end-to-end testing) and our Year 2000 test 

guidance3

 define a number of key management controls to employ when 

planning, executing, analyzing, and reporting on such test and evaluation 

events. 

We found that Space Command’s space control operational evaluation 

satisfied 16 of 21 of the key processes prescribed by JCS guidance. For 

example, the Command established a Y2K task force to guide the 

evaluation effort, which included satellite/system specialists, test and 

evaluation experts, system analysts, military component and service 

representatives, and public affairs representatives. Further, the Command 

performed a rehearsal before conducting the evaluation to ensure that all 

critical systems and interfaces were operating correctly and that all staff 

knew their roles and responsibilities.

In response to our concerns, Space Command has taken positive actions to 

address the remaining five key processes. Three of the key processes were 

addressed during the course of our review and two were addressed in 

response to a recommendation we made at our briefing. During the course 

of our review, Space Command began ensuring that contingency plans 

were in place for its mission-critical systems, which it had not done before 

conducting the space control operational evaluation. Also, after we found 

that configuration management procedures were not always followed

2

Joint Staff Year 2000 Operational Evaluation Guide, Version 3.0, April 1, 1999.

3

Year 2000 Computing Crisis: A Testing Guide (GAO/AIMD-10.1.21, issued as an exposure 

draft in June 1998; issued in final in November 1998).
B-282546

Page 5 GAO/AIMD-00-30 Space Command Y2K Operational Testing

while executing the evaluation,4 Space Command initiated an effort to 

ensure that such procedures are followed in future evaluations. In addition, 

during our review, the Command amended its report to discuss its decision 

to exclude six communications systems from the evaluation and whether 

this adversely impacted the ability to draw conclusions about mission 

readiness.

At the time of our October 1, 1999, briefing, Space Command still needed to 

address two partially satisfied key processes, which included (1) not 

documenting whether test cases for most intelligence systems met 

performance exit criteria and (2) not ensuring that 1 of 29 systems included 

in the evaluation was Y2K compliant. We therefore recommended that 

Space Command amend its final report to JCS to recognize the 

uncertainties and risks associated with its failure to take these steps and 

the actions underway or planned to address these uncertainties and risks. 

Without taking these steps, Space Command could not adequately know 

the Year 2000 readiness of critical tasks—collecting surveillance and 

intelligence data to disseminate warning messages—associated with 

conducting the space control mission. Because Space Command has 

subsequently amended its final report and plans to ensure that these 

weaknesses are not repeated in a November operational evaluation of its 

intelligence mission, we are not making further recommendations at this 

time. 

Background Space Command’s mission is to provide direct support to combatant 

commanders and military forces through the use of space-based satellites 

and other technologies needed for navigation, surveillance and 

reconnaissance, communications, environmental and attack warnings 

during war and peacetime operations. To perform this mission, Space 

Command relies on a wide array of information technology systems, 

including command and control systems, geographically dispersed radar 

sites, satellites, communications networks, and intelligence systems. 

In August 1998, the Secretary of Defense directed JCS to require its 

combatant commands, including Space Command, to plan, execute, 

4

Configuration management involves establishing product baselines and systematically 

controlling changes made to those baselines. Without an effective configuration 

management process, organizations can lose control of the software product, potentially 

produce and use inconsistent product versions, and create operational problems.
B-282546

Page 6 GAO/AIMD-00-30 Space Command Y2K Operational Testing

analyze, and report on a series of simulated Year 2000 operational 

evaluations. The evaluations, which were to assess whether DOD can 

continue to perform critical military operations in a Year 2000 environment, 

are one of three DOD end-to-end testing efforts.5

The purpose of end-to-end testing is to verify that a defined set of 

interrelated systems, which collectively support an organizational core 

business area or function, interoperate as intended in an operational 

environment (either actual or simulated). These interrelated systems 

include not only those owned and managed by an organization, but also the 

external systems with which they interface or that otherwise support the 

business area or function. The combatant commands’ core business areas 

or functions are referred to as “thin lines.”

The boundaries for end-to-end tests can vary depending on a given business 

function’s system dependencies and criticality to the organizational 

mission. Therefore, in managing end-to-end test activities, it is important to 

analyze the interrelationships among core business functions and their 

supporting systems and the mission impact and risk of date-induced system 

failures and to use these analyses to define test boundaries. It is also 

important to work early and continually with functional partners to ensure 

that related end-to-end test activities are effectively coordinated and 

integrated. Table 1 summarizes key processes recommended by JCS’ Year 

2000 operational evaluation guidance, which is consistent with our Year 

2000 test guide.

5

In addition to conducting operational evaluations, the military services are conducting 

system integration testing, and the functional business areas, such as personnel and health 

affairs, are conducting functional end-to-end tests. Each of these end-to-end testing 

activities is discussed in detail in Defense Computers: Management Controls Are Critical to 

Effective Year 2000 Testing (GAO/AIMD-99-172, June 30, 1999).
B-282546

Page 7 GAO/AIMD-00-30 Space Command Y2K Operational Testing

Table 1: Summary of JCS Year 2000 Operational Evaluation Criteria

Space Command has already completed 16 operational evaluations to 

assess its ability to manage and provide combatant support during a major 

theater war. These evaluations covered seven mission areas, including

(1) integrated tactical warning and attack assessment, (2) space control, 

(3) force enhancement, (4) weather support, (5) command and control of 

space forces, (6) space operations support, and (7) space lift. The space 

control mission area provides (1) surveillance support to monitor, track, 

identify, and catalog all orbiting space objects for collision avoidance and 

(2) protection support to monitor, detect, assess, characterize, track, and 

issue warnings about threats, both natural and man-made, against United 

Planning • Specify test assumptions and limitations

• Establish a Year 2000 task force

• Identify critical missions/tasks/systems

• Verify that systems essential to mission are Year 2000 compliant

• Develop an operational evaluation plan to guide event planning and 

execution

• Identify and schedule support from other commands, DOD components, 

etc.

• Determine relevant and necessary resources (e.g., funding, personnel, 

equipment, etc.)

• Ensure approved Year 2000 contingency plans are prepared

• Develop a risk management plan

• Identify simulation needs and establish supporting testing environment

• Develop data collection and analysis plan or approaches

Execution • Conduct operational evaluation rehearsal

• Follow configuration management policy

• Perform baseline test for operational evaluation

• Execute required Year 2000 date rollover tests

• Collect and archive al Year 2000-relevant data and ensure that systems 

are reset to current day operations

Analysis • Categorize, document, and report Year 2000 failures

• Determine mission impact of Year 2000 failures

• Ensure exit criteria are met

Reporting • Prepare Year 2000 reports describing mission impact and readiness

• Provide reports to JCS within required timeframes
B-282546

Page 8 GAO/AIMD-00-30 Space Command Y2K Operational Testing

States and allied space systems. The space control evaluation was 

executed between March 11 and March 25, 1999.

Space Command 

Implemented Most 

Important Management 

Processes During Its 

Space Control 

Evaluation

As noted in table 2 below, we found that, for its space control operational 

evaluation, Space Command satisfied the majority of the management 

process controls (16 of 21) specified in JCS’ operational evaluation 

guidance. 

Table 2: Summary of Space Command’s Satisfaction of JCS Evaluation Criteria for 

the Space Control Evaluation

Consistent with JCS guidance governing operational evaluation planning, 

Space Command established a Year 2000 task force, which included 

satellite/system specialists, test and evaluation experts, system analysts, 

military component and service representatives, and public affairs 

representatives. It identified 35 critical tasks that it needed to carry out the 

space control mission in support of a major theater war. Space Command 

also issued a directive to ensure testing resources would be made available 

for operational evaluations and earmarked about $8 million for operational 

evaluation activities—including the space control evaluation. Further, 

Space Command developed a test plan that documented participant roles 

and responsibilities, critical missions and tasks, test cases, and reporting 

requirements.

Space Command also took effective steps in executing, analyzing, and 

reporting on its evaluation. For instance, before executing the operational 

evaluation, Space Command performed a rehearsal to ensure that all 

critical systems and interfaces were operating correctly and that all staff 

knew their roles and responsibilities. Before resetting systems to current 

day operations, Space Command ensured that thin line systems were 

Phases

Number of primary

evaluation criteria

Number of primary

criteria satisfied

Planning 11 9

Execution 5 4

Analysis 3 2

Reporting 2 1

Total 21 16
B-282546

Page 9 GAO/AIMD-00-30 Space Command Y2K Operational Testing

assessed, master scenario events were performed and deviations were 

identified, and that all data needed to make an assessment of the 

command’s ability to perform the space control mission were collected and 

archived. 

Space Command Acted 

to Address Three 

Partially Satisfied Key 

Processes

Following its operational evaluation, Space Command took action to 

resolve three partially satisfied key processes. In doing so, it increased its 

assurance with respect to the Y2K readiness of space control critical tasks 

involving intelligence and communications systems.

First, before conducting its test, Space Command did not verify that 

contingency plans were in place for the 29 systems included in the 

evaluation. Instead, Space Command relied exclusively on system owners 

to do so. As noted in JCS testing guidance, contingency plans identify 

alternative systems or workaround procedures to use when performing a 

mission in the event of a system disruption. As such, JCS guidance states 

that it is essential that commands ensure that these plans are in place prior 

to executing the operational evaluation so that they can be invoked in the 

case of system failure. Subsequent to the evaluation, Space Command 

began verifying that contingency plans are in place for its mission-critical 

systems.

Second, while executing the evaluation, Space Command did not follow 

configuration management procedures. JCS guidance specifies that system 

configurations not be changed during testing unless authorized by the test 

director. During the space control evaluation, changes were made to one 

system after the baseline for the evaluation was established and without 

authorization from the test director. These changes contributed to a “hard” 

failure during testing. (Information on the nature of the system failure is 

classified).6

 After the evaluation, Space Command directed the 17th Test 

Squadron and intelligence unit to review this deviation and its impact on 

the command’s ability to determine mission readiness. On September 30, 

1999, the intelligence unit and 17th Test Squadron reported that the 

deviation did not materially affect mission readiness. To prevent similar 

problems in future evaluations, Space Command directed the 17th Test 

Squadron and intelligence unit to develop ways to improve testing 

6

A “hard” failure is a Y2K-related failure that results in an obvious adverse impact to the 

system. For example, the system shuts down, displays erroneous data, or performs other 

unexpected actions.
B-282546

Page 10 GAO/AIMD-00-30 Space Command Y2K Operational Testing

documentation and procedures with a special focus on ensuring that 

documentation standards, configuration management procedures, and 

baseline test requirements are followed. 

Third, in reporting on the evaluation, Space Command did not specify how 

its exclusion of six communications systems from the test impacted its 

ability to draw conclusions about mission readiness. When planning the 

evaluation, Space Command concluded that it would not include six 

communications systems in the evaluation due to resource constraints or 

because the systems were to be included in a future evaluation. As a result, 

Space Command assumed that communications systems would be 

available to perform critical tasks and disseminate time-sensitive warnings 

to combatant commanders. While Space Command communicated this 

assumption to JCS in its operational evaluation plan, it did not report on 

how this scope limitation could adversely affect its ability to draw 

conclusions about mission readiness. Instead, Space Command reported to 

JCS that critical space control tasks could be performed across the 

calendar and leap year dates with no significant impact on its mission 

readiness. Space Command has since ensured that omitted 

communications systems were included in other Year 2000 end-to-end 

testing or operational evaluation events and disclosed this limitation in its 

final report on the evaluation.

Space Command Is 

Acting to Address 

Recommendation 

Made at the Briefing

At the time of our October 1, 1999, briefing, Space Command had not yet 

addressed two partially satisfied key processes. First, in planning the 

evaluation, Space Command did not ensure that one intelligence system to 

be tested was certified as compliant. Rather, it only verified that the 

software application relevant to the evaluation was compliant. Year 2000 

compliance of an application in isolation is of very limited value unless the 

system platform that it runs on, as well as other applications operating on 

the system, is compliant. As such, both JCS guidance and GAO’s end-to-end 

test guidance define system, not application, compliance as a precondition 

to end-to-end testing. 

Second, Space Command did not document whether intelligence systems 

met system performance exit criteria for all test cases. Specifically, the 

command was supposed to show whether it could process a predetermined 

number of transactions within specific time constraints. While command 

officials contend that this was done, only one-fifth of the transactions for 

intelligence critical tasks were documented. Space Command officials 
B-282546

Page 11 GAO/AIMD-00-30 Space Command Y2K Operational Testing

stated that it was too time-consuming for operators to print screens for 

these tasks during the evaluation. 

At our briefing, we recommended that Space Command amend its final 

report to JCS to recognize the (1) uncertainties and risks associated with 

its failure to fully satisfy these criteria and (2) the actions it had underway 

or planned to address these uncertainties and risks. Space Command 

agreed with this recommendation. It plans to amend its final report to 

disclose these limitations and to pursue an alternative data collection 

strategy for its planned November 1999 operational evaluation of its 

intelligence mission in order to verify that intelligence systems/tasks fully 

meet performance criteria.

Conclusion By acting swiftly to address our recommendation, made during the

October 1, 1999, briefing, Space Command has demonstrated its 

commitment to improving management controls over Year 2000 testing 

activities and the effectiveness and value of its operational evaluation as 

well as mitigated the risks associated with being able to operate effectively 

in the Year 2000. Further, it has ensured that DOD managers have complete 

and reliable information to use in making informed military decisions. As a 

result, Space Command has satisfied the intent of our recommendation, 

and we are not making any further recommendations at this time.

We are sending copies of this report to Representative John P. Murtha, 

Ranking Minority Member, Subcommittee on Defense, House 

Appropriations Committee; Senator John Warner, Chairman, and Senator 

Carl Levin, Ranking Minority Member, Senate Committee on Armed 

Services; Senator Ted Stevens, Chairman, and Senator Daniel Inouye, 

Ranking Minority Member, Subcommittee on Defense, Senate Committee 

on Appropriations; and Representative Floyd Spence, Chairman, and Ike 

Skelton, Ranking Minority Member, House Committee on Armed Services.

We are also sending copies to the Honorable John Koskinen, Chair of the 

President’s Year 2000 Conversion Council; the Honorable William Cohen, 

Secretary of Defense; the Honorable John Hamre, Deputy Secretary of 

Defense; General Henry Shelton, Chairman of the Joint Chiefs of Staff; 

Arthur Money, Assistant Secretary of Defense for Command, Control, 

Communications, and Intelligence; and the Honorable Jacob Lew, Director, 

Office of Management and Budget. Copies will also be made available to 

others upon request.
B-282546

Page 12 GAO/AIMD-00-30 Space Command Y2K Operational Testing

Should you or your staff have any questions concerning this report, please 

contact me at (202) 512-6240. I can also be reached by e-mail at 

brockj.aimd@gao.gov. Other points of contact and key contributors to this 

report are listed in appendix III.

Sincerely yours,

Jack L. Brock, Jr.

Director, Governmentwide and Defense

 Information Systems
Page 13 GAO/AIMD-00-30 Space Command Y2K Operational Testing
Page 14 GAO/AIMD-00-30 Space Command Y2K Operational Testing

Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL ApendxIi

1

Results of GAO Review of

 SPACECOM Space Control Y2K OPEVAL

House Appropriations Committee

November 1, 1999
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 15 GAO/AIMD-00-30 Space Command Y2K Operational Testing

2

Introduction

• In August 1998, the Secretary of Defense directed the Commandersin-Chief (CINC), who are responsible for Defense’s unified combatant

commands, to plan, execute, and report on a series of simulated Year

2000 operational evaluations (Y2K OPEVALs).

• The CINC Y2K OPEVALs are one of three Defense Y2K end-to-end

test and evaluation efforts. GAO’s Y2K Test Guide advocates end-toend testing, which is testing performed to verify that a defined set of

interrelated systems (i.e., systems that collectively support an

organizational core business function or operation) interoperate as

intended in a Y2K environment.

The CINC core business functions/operations are referred to as “thin

lines.” The “thin lines” consist of critical tasks, as well as systems that

perform critical tasks.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 16 GAO/AIMD-00-30 Space Command Y2K Operational Testing

3

Objectives

• At the request of the Chairman, House Appropriations Committee,

Defense Subcommittee, GAO is reviewing selected OPEVALs to

determine:

(1) if the OPEVAL was planned, executed, and documented in

accordance with DOD guidelines, and

(2) what the OPEVAL results indicated concerning readiness and risks.

• The OPEVALS reviewed by GAO included those conducted by

Space and Transportation Commands and were selected in

collaboration with the Defense Inspector General (IG) to ensure:

– appropriate coverage of all CINC OPEVALs, and

– no duplication of effort.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 17 GAO/AIMD-00-30 Space Command Y2K Operational Testing

4

Scope and Methodology

• reviewed the OPEVAL plan, testing documents/records, and test

results/reports;

• interviewed Space Command officials responsible for Y2K

OPEVAL planning, execution, and reporting tasks; and

• compared Space Command’s planning, execution, analysis, and

reporting actions with Defense OPEVAL guidance.

This briefing addresses the Space Command OPEVAL for Space

Control. To satisfy objective (1), we
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 18 GAO/AIMD-00-30 Space Command Y2K Operational Testing

5

Scope and Methodology

• To satisfy objective (2), we

• reviewed Space Command’s OPEVAL results, 7-and 30-day

reports, and system problem tracking reports and

• interviewed Space Command officials and analysts responsible for

developing OPEVAL assessment methodologies, interpreting

evaluation metrics, and ensuring that evaluation exit criteria were

met.

• On October 1, 1999, we briefed Space Command leadership on the

results of our review and made a recommendation to address our

findings. In agreeing to our recommendation, Space Command has

taken action to address weaknesses identified during the review and

plans to amend its final OPEVAL report accordingly.

• We performed our work from March 1999 through October 1999 in

accordance with generally accepted government auditing standards.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 19 GAO/AIMD-00-30 Space Command Y2K Operational Testing

6

Background

• To assist the CINCs in planning, documenting, executing,

analyzing, and reporting OPEVALs, the Joint Staff issued OPEVAL

guidance. The guidance is divided into phases:

– planning,

– execution,

– analysis, and

– reporting

• The OPEVAL guidance is consistent with GAO’s end-to-end testing

guidance and DOD’s Y2K management plan.

DOD OPEVAL Overview
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 20 GAO/AIMD-00-30 Space Command Y2K Operational Testing

7

Background

• Space Command is responsible for providing continuous and real-time

 (1) warnings of air or space attacks against North America and (2) space

control, surveillance, communications, and intelligence support to military

operations worldwide.

• To fulfill its missions, Space Command depends heavily on information

systems and technology, including satellites, geographically dispersed

radars/sensors, ground relay terminals/stations, and communication

networks.

Space Command OPEVAL Events’ Status
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 21 GAO/AIMD-00-30 Space Command Y2K Operational Testing

8

Background

• Space Command identified 7 “thin lines” (missions) to be

operationally evaluated: (1) Integrated Tactical Warning and Attack

Assessment, (2) Space Control, (3) Force Enhancement,

 (4) Weather Support, (5) Command and Control (C2) of Space

Forces, (6) Space Operations Support, and (7) Space Lift.

• For the 7 “thin lines”, Space Command identified 92 critical tasks1

and 86 supporting systems.

• The following table describes the status of Space Command’s 16

OPEVALs and the Chairman’s Contingency Assessment (CCA).

1 The number does not include critical tasks for the first two OPEVALS because critical tasks were not

identified in reports for the first two OPEVALS.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 22 GAO/AIMD-00-30 Space Command Y2K Operational Testing

9

Background

Evaluation Events “Thin Line” Schedule Results

OPEVAL – North

American

Aerospace Defense

Command

(NORAD)

Integrated Tactical

Warning and Attack

Assessment

(ITWAA)

December 2-4,

1998

Reported mission ready

0 “soft”1 failures

0 “hard” 2 failures

OPEVAL – NORAD ITWAA February 16-28,

1999

Reported mission ready

2 “soft” failures

1 “hard” failure

OPEVAL Space Control March 15-25,

1999

Reported mission ready

2 “soft”1 failures

3 “hard”2 failures

OPEVAL – Central

Command

Command and

Control (C2) –

Space Forces

April 6-12, 1999 Reported mission ready

1 “soft”1 failure

0 “hard”2 failures

 1A “soft” failure is a Y2K-related failure that is not immediately discernable. The effect may be cumulative

and require several hours, days, or longer to manifest itself. 2

A “hard” failure is a Y2K-related failure that results in an obvious adverse impact to the system. For

example, the system shuts down, erroneous data is displayed, or unexpected actions occur. 3 Information classified by the Department of Defense.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 23 GAO/AIMD-00-30 Space Command Y2K Operational Testing

10

Background

Evaluation Events “Thin Line” Schedule Results

OPEVAL – Central

Command

Global Positioning

System - Space

Support/Space

Force Enhancement

April 23-May 1,

1999

Reported mission ready

0 “soft” failures

0 “hard” failures

OPEVAL – Central

Command

Theater Ballistic

Missile Warning

(TBMW) – Space

Support/Space

Force Enhancement

April 22-30,

1999

Reported mission ready

0 “soft” failures

1 “hard” failure

OPEVAL – Central

Command

Space Support/

Satellite Control

May 1-2 &

May 10-11,

1999

Reported mission ready

3 “soft”1 failures

0 “hard”2 failures

OPEVAL – Central

Command

Terrestrial Weather

- Space Support/

Space Force

Enhancement

May 10-17,

1999

Reported mission ready

1 “soft”1 failure

0 “hard”2 failures
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 24 GAO/AIMD-00-30 Space Command Y2K Operational Testing

11

Background

Evaluation Events “Thin Line” Schedule Results

OPEVAL Communications –

Space Operations

Support

June 9-15, 1999 Reported mission ready

0 “soft”1 failures

0 “hard”2 failures

CCA4 Intelligence,

Surveillance, &

Reconnaissance

June 14-18,1999 Classified3

OPEVAL Space Weather June 19-July 14, 1999 Reported mission ready

0 “soft”1 failures

0 “hard”2 failures

OPEVAL –

Central Command

TBMW – C2 Space

Forces

July 15-31, 1999 Reported mission ready

0 “soft” failures

1 “hard” failure

OPEVAL – Central

Command

Communications –

Space Force

Enhancement

July 27, 1999 Reported mission ready

1 “soft” failure

0 “hard failures

4

The CCA was designed to evaluate the ability of unified commands to perform missions in an

environment degraded by Y2K failures.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 25 GAO/AIMD-00-30 Space Command Y2K Operational Testing

12

Background

Evaluation Events “Thin Line” Schedule Results

OPEVAL Space Lift July 22, 1999 Reported mission ready

0 “soft” failures

0 “hard” failures

OPEVAL Space Lift July 27, 1999 Reported mission ready

0 “soft” failures

0 “hard” failures

OPEVAL Space Lift August 2-6, 1999 Reported mission ready

0 “soft” failures

0 “hard” failures

OPEVAL Space Lift September 1-3, 1999 Reported mission ready

0 “soft” failures

0 “hard” failures
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 26 GAO/AIMD-00-30 Space Command Y2K Operational Testing

13

Background

• The Space Control “thin line” includes providing (1) surveillance

support to monitor, track, identify, and catalog all orbiting space

objects for collision avoidance and (2) protection support to monitor,

detect, assess, characterize, track, and issue warnings about threats,

both natural and man-made, against U.S. and allied space systems.

• The Space Control OPEVAL was conducted in collaboration with

other DOD organizations, including Air Force, Army, and Navy Space

Commands. It was intended to test real-world Space Control

operations in a Y2K environment.

Space Control OPEVAL
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 27 GAO/AIMD-00-30 Space Command Y2K Operational Testing

14

Background

• The Space Control “thin line” included 50 systems supporting 35 critical

tasks. The OPEVAL involved 29 of these systems.

• The Space Control OPEVAL was completed between March 11-25, 1999

and included these test execution events:

– Rehearsal: March 11, 1999.

– Baseline: March 20, 1999.

– Surveillance/Intelligence Testing: March 15-19, 1999.

– Protection Testing: March 22-25, 1999.

• The test environment consisted of desktop computers; geographically

dispersed ground radar sites; and partitioned IBM mainframes.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 28 GAO/AIMD-00-30 Space Command Y2K Operational Testing

15

Background

• The required calendar and leap year events (September 8, 1999 to

September 9, 1999; December 31, 1999 to January 1, 2000; February

28, 2000 to February 29, 2000; and February 29, 2000 to March 1, 2000)

were assessed as part of the OPEVAL.

• The Space Control OPEVAL assessed 46 date dependent functions in

29 systems using these dates.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 29 GAO/AIMD-00-30 Space Command Y2K Operational Testing

16

Background

Space Control Critical Tasks Thin Line Systems

1. Maintain the current space environment

database

Navy Fence; Millstone; Thule; TOS; Altair;

MPDS; IDHS; SPADOC; CMP; CFE-R;

ICIG; NUIS; AMHS

2. Space Surveillance Network tasking MPDS; SPADOC; CMP

3. Observation control/tasking analysis MPDS; SPADOC; CMP

4. Observation Processing SPADOC

5. Element Set Updates SPADOC

6. Transmit Field Element Sets SPADOC

7. Cross-tag/lost satellite/unknown observation

processing

SPADOC

8. Launch processing SPADOC
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 30 GAO/AIMD-00-30 Space Command Y2K Operational Testing

17

Background

Space Control Critical Tasks Thin Line Systems

 9. Maneuver Processing SPADOC

10. Manual Piece Separation Processing SPADOC

11. Collision Avoidance SPADOC

12. Decay Processing/re-entry assessment SPADOC

13. Break-up Processing SPADOC

14. Monitor and report status of all sensor sites C MPDS; SPADOC; CMP

15. Manage sensor coverage MPDS; SPADOC; CMP

16. Perform event-related up-channel reporting MPDS; SPADOC; CMP
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 31 GAO/AIMD-00-30 Space Command Y2K Operational Testing

18

Background

Space Control Critical Tasks Thin Line Systems

 17. Perform extended collection/surveillance

against foreign satellites to characterize status and

performance parameters as well as support the

foreign space order of battle

Navy Fence, Millstone, Thule, TOS, Altair;

MPDS; IDHS; SPADOC; CMP; CFE-R;

ICIG; NUIS; AMHS; OSAS; SDB; SMAT;

SMPAS

18. Monitor the space situation and collect and

correlate data on potential and actual hostile

activities against U.S. and designated allied space

systems.

Navy Fence, Millstone, Thule, TOS, Altair;

MPDS; IDHS; SPADOC; CMP; CFE-R;

ICIG; NUIS; AMHS; SDB; SATRAN

19. Assess data and determine intent SPADOC; AMHS; NUIS; OSAS; SDB;

SMAT; SMPAS

20. Inform the National Military Command Center of

impending, current, and completed hostile space

activity

MPDS; SPADOC; CMP

21. Characterize the results of a space attack SPADOC
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 32 GAO/AIMD-00-30 Space Command Y2K Operational Testing

19

Background

Space Control Critical Tasks Thin Line Systems

22. Provide designated authorities with situation

reports

MPDS; SPADOC; CMP

23. Provide technical support as required MPDS; SPADOC; CMP

24. Provide assistance to routine peacetime space

operations

MPDS; SPADOC; CMP

25. Provide warning and assessment messages MPDS; SPADOC; CMP

26. Provide countermeasure coordination/status MPDS; SPADOC; CMP

27. Inform space system owner/operators and other

designated authorities of selected countermeasures

MPDS; SPADOC; CMP
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 33 GAO/AIMD-00-30 Space Command Y2K Operational Testing

20

Background

Space Control Critical Tasks Thin Line Systems

28. Evaluate countermeasure effectiveness SPADOC

29. Assist in planning for further countermeasures SPADOC

30. Inform National Military Command Center and

other appropriate elements of the results of

countermeasure implementation

MPDS; SPADOC; CMP

31. Maintain current documentation of world-wide

counter space capabilities

SPADOC; CFE-R; ICIG; NUIS; SDB;

SMPAS; OSAS; SMAT

32. Receive taskings (sensors) Navy Fence; Millstone; Thule; TOS; Altair;

MPDS; CMP

33. Schedule tracks (sensors) Navy Fence; Millstone; Thule; TOS; Altair

34. Conduct tracks (sensors) Navy Fence; Millstone; Thule; TOS; Altair

35. Transmit track data (sensors) Navy Fence; Millstone; Thule; TOS; Altair;

MPDS; IDHS
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 34 GAO/AIMD-00-30 Space Command Y2K Operational Testing

21

Results of GAO Review
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 35 GAO/AIMD-00-30 Space Command Y2K Operational Testing

22

Planning

Defense Test Criteria Result

Specify test assumptions and limitations Satisfied

Establish a Y2K task force and assign responsibilities Satisfied

Identify critical missions/tasks/systems Satisfied

Verify systems essential to mission are Y2K compliant/certified Partially Satisfied

Develop OPEVAL plan to guide event planning and execution Satisfied

Identify and schedule CINC/Allied/Component/Agency support Satisfied

Determine relevant and necessary resources (e.g., funding,

personnel, equipment, etc.)

Satisfied

Ensure approved Y2K contingency plans are prepared Partially Satisfied

Develop risk management plan Satisfied

Identify simulation needs and establish supporting environment Satisfied

Develop data collection and analysis plan or approaches Satisfied
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 36 GAO/AIMD-00-30 Space Command Y2K Operational Testing

23

Findings: Planning

Criteria: In planning for the OPEVAL, CINCs are to define assumptions

concerning the readiness of systems and the ability to evaluate systems in

light of real-world limitations.

Finding: Space Command identified real-world considerations and system

readiness limitations during Y2K planning meetings. These limitations

were disclosed in the OPEVAL Plan and to the JCS. Specifically, Space

Command reported that 29 of the 50 “thin line” systems would be included

and 21 would be excluded from the OPEVAL due to resource constraints,

or because they would be tested in other OPEVALs. Six of the 21

excluded systems were communications systems that were to be tested in

a future OPEVAL.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 37 GAO/AIMD-00-30 Space Command Y2K Operational Testing

24

Findings: Planning

Criteria: A CINC Y2K Task Force composed of knowledgeable

Y2K, test, and systems experts should be formed to establish

the base for all Y2K planning, coordination, execution, and

reporting.

Finding: Consistent with the defined scope of the OPEVAL,

Space Command established a Y2K Task Force and it defined

roles and responsibilities with milestones for each member.

Members of the task force included satellite/system specialists,

test and evaluation experts, system analysts, and public affair

specialists from the Command’s Operations, Intelligence,

Planning, and Public Affairs units. They also included Air

Force’s Operational Test and Evaluation Center (AFOTEC) and

17th Test Squadron, military service, and NASA

representatives.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 38 GAO/AIMD-00-30 Space Command Y2K Operational Testing

25

Findings: Planning

Criteria: CINCs need to analyze critical missions to determine

the most critical missions and identify the critical tasks

supporting each critical mission. In addition, the minimum

number of integrated automated information platforms/systems

required to perform each critical task or critical mission must be

identified (the “thin line”).

Finding: Consistent with the defined scope of the OPEVAL,

Space Command identified 35 critical tasks that needed to be

evaluated to determine mission readiness in a Y2K

environment. In addition, Space Command identified a total of

29 “thin line” systems to support these tasks.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 39 GAO/AIMD-00-30 Space Command Y2K Operational Testing

26

Findings: Planning

Criteria: Ensure that mission-critical “thin line” systems are

certified Y2K compliant.

 Finding: Consistent with the defined scope of the OPEVAL,

Space Command verified that 28 of 29 mission-critical, “thin

line” systems to be included in the OPEVAL were certified Y2K

compliant. The 29th system was not certified as compliant, but

was nevertheless included in the OPEVAL rather than invoking

the system’s contingency plan because, according to Space

Command officials, they verified that the application on the

system relevant to the OPEVAL was compliant. This is contrary

to JCS guidance and GAO’s end-to-end test guidance, which

defines system (not application) compliance as a precondition

to end-to-end testing. In short, Y2K compliance of an

application in isolation is of very limited value unless the system

platform that it runs on, and the other applications running on

the system that it interoperates with, are also compliant.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 40 GAO/AIMD-00-30 Space Command Y2K Operational Testing

27

Finding: Planning

Finding: At the time of our October 1, 1999, briefing, Space

Command had not addressed this concern. We therefore

recommended that this deviation be disclosed in the final

OPEVAL report. Space Command officials agreed with our

recommendation that the final report disclose this information

and now plans to revise the final report. Additionally, Space

Command stated that the system is scheduled to be

compliant in November 1999 and to be included in its

November 1999 operational evaluation of the intelligence

mission area.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 41 GAO/AIMD-00-30 Space Command Y2K Operational Testing

28

Findings: Planning

Criteria: The Y2K task force should document how the OPEVAL

will be conducted, data will be gathered and analyzed, and how

reports will be formatted.

Finding: Space Command developed an exercise directive and

test plan for the OPEVAL to:

– ensure that mechanisms for evaluating critical dates and contingency

plans for mission-critical systems are executed.

– document participant roles and responsibilities.

– link critical missions, critical tasks, architectures, test cases, and data

elements.

– report Y2K OPEVAL results.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 42 GAO/AIMD-00-30 Space Command Y2K Operational Testing

29

Findings: Planning

Criteria: When preparing for a Y2K OPEVAL, determine the

extent of participation of other CINCs, allies, components, and

agencies and coordinate their participation in the event.

Finding: Consistent with the defined scope of the OPEVAL,

Space Command identified, coordinated, and scheduled the

OPEVAL with Y2K Task Force members from the Command’s

Operations, Intelligence, Planning, and Public Affairs units.

They also coordinated activities with Air Force’s Operational

Test and Evaluation Center (AFOTEC), 17th Test Squadron,

and Air Force, Army, and Navy Space Commands, and NASA.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 43 GAO/AIMD-00-30 Space Command Y2K Operational Testing

30

Findings: Planning

Criteria: The necessary resources (funding, personnel, training,

equipment, time frames, and external organization support)

should be identified and included in the plan.

Finding: In November 1998, Space Command issued a directive

to ensure testing resources would be made available for Y2K

OPEVALS. About $8 million was earmarked for OPEVAL

activities, including the Space Control OPEVAL. Space

Command also coordinated the evaluation scenario and scripts

with all OPEVAL participants, acquired the systems hardware

and software to simulate space control events, and scheduled

37 test and operator personnel to help execute the OPEVAL.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 44 GAO/AIMD-00-30 Space Command Y2K Operational Testing

31

Findings: Planning

Criteria: To ensure that Y2K exercise objectives are met, it is

essential to have contingency plans in place prior to executing

the OPEVAL.

Finding 1: Space Command’s approach to determining whether

contingency plans were in place prior to executing the OPEVAL

was to rely exclusively on system owners to ensure that this

criterion was met. Space Command did not take steps to verify

this criterion. Space Command has since initiated a review of

about 50 contingency plans.

Finding 2: During the OPEVAL, operators successfully

performed mission tasks in response to the 3 “hard” system

failures by invoking workarounds.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 45 GAO/AIMD-00-30 Space Command Y2K Operational Testing

32

Findings: Planning

Criteria: CINC-unique risk management plans should be developed to

identify and mitigate system-related risks before they adversely

impact mission execution.

Finding: Space Command identified OPEVAL risks and strategies for

managing these risks in its Space Control OPEVAL Plan. For

example, the Plan recognizes the risks associated with confusing

OPEVAL sensor observations with real-world observations. To

mitigate these risks, Space Command’s Plan provides strategies for

isolating systems’ execution of OPEVAL tasks/functions from realworld system operations by electronic partitioning.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 46 GAO/AIMD-00-30 Space Command Y2K Operational Testing

33

Findings: Planning

Criteria: CINCs should (1) determine if simulations or manual

data input will be needed during the execution of the OPEVAL,

and, if needed, (2) ensure that an environment which can

support the simulation is planned for and acquired.

Finding: Within the defined scope of the OPEVAL, Space

Command identified the simulations needed and manual data

inputs required for testing and ensured that data injection

methodologies were included in the OPEVAL Plan and master

scenario events list (MSEL). For example, Space Command

used simulated scenarios to perform satellite orbital changes

for tracking and cataloging purposes.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 47 GAO/AIMD-00-30 Space Command Y2K Operational Testing

34

Findings: Planning

Criteria: A plan should be prepared to help coordinate and

synchronize all OPEVAL data collection and assessment

activities.

Finding: Consistent with the specified scope of the OPEVAL,

Space Command developed a data collection and analysis plan

that included (1) specific actions that should be accomplished by

the OPEVAL participants prior to the start of and at the

completion of each OPEVAL, (2) ground rules for collecting and

documenting mission-critical system outputs, and (3) direction

on reviewing the critical tasks executed during the OPEVAL and

determining the performance of the mission-critical “thin line”

systems.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 48 GAO/AIMD-00-30 Space Command Y2K Operational Testing

35

Execution

Defense Test Criteria Result

Conduct OPEVAL rehearsal Satisfied

Follow configuration management policies Partially Satisfied

Perform baseline test for OPEVAL Satisfied

Execute required Y2K date rollover tests Satisfied

Collect and archive all Y2K-relevant data and ensure that

systems are reset to current day operations Satisfied
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 49 GAO/AIMD-00-30 Space Command Y2K Operational Testing

36

Findings: Execution

Criteria: Prior to executing the Y2K OPEVAL, a rehearsal should be

conducted to ensure that all critical systems and interfaces identified

in the system architecture are operating correctly and that OPEVAL

staff know their roles and responsibilities.

Finding: Space Command performed a rehearsal/test readiness

review on March 11, 1999. The rehearsal was used to (1) validate

that the test readiness review requirements, (2) verify data collection

and analysis methodologies, (3) confirm the baseline configuration

for testing, and (4) ensure OPEVAL staff practiced their roles and

responsibilities.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 50 GAO/AIMD-00-30 Space Command Y2K Operational Testing

37

Findings: Execution

Criteria: The configurations of systems and architecture

established for OPEVAL testing should not be changed unless

authorized by the test director.

Finding: Changes were made to one system in the test

environment after the baseline for the OPEVAL was established

and without authorization from the test director. These changes

contributed to one “hard” system failure identified during the

OPEVAL. (Information on the nature of the system failure is

classified.) Space Command has since reinforced the need to

strictly follow configuration management policies during

OPEVALS and tasked the 17th Test Squadron and intelligence

unit to develop ways to better ensure that configuration

management over test baselines is enforced.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 51 GAO/AIMD-00-30 Space Command Y2K Operational Testing

38

Findings: Execution

Criteria: A baseline Y2K test should be executed to establish expected

results data that will be used to compare to output data captured during the

Y2K date rollover tests and to help establish whether or not a failure is

Y2K-related.

Finding: Space Command conducted a baseline Y2K test on

March 20, 1999. However, this test only covered the critical tasks

associated with space surveillance and protection and did not include

intelligence tasks because officials stated that baseline testing duplicated

rehearsal activities. This position is contrary to JCS guidance. According to

JCS guidance, the purpose of the rehearsal is to provide operators with an

opportunity to practice their responsibilities. In contrast, the baseline test is

to execute the master scenario events list under operational conditions to

establish expected outputs against which OPEVAL results can be

compared. Clearly, these two execution requirements, because they serve

different purposes, differ in terms of content, depth, and scope, and thus

are not duplicative.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 52 GAO/AIMD-00-30 Space Command Y2K Operational Testing

39

Findings: Execution

Finding: Space Command officials acknowledge the

differences between test rehearsals and baseline tests, but

explained that the rehearsal for intelligence systems was

expanded to satisfy baseline testing requirements,

including using the same quality and quantity of data

planned for the baseline test. To verify this, we reviewed

information subsequently provided by the test directorate

and found that it showed a level of testing rigor for these

systems that went beyond that normally required of a test

rehearsal. We also verified that baseline test results were

documented during the rehearsal. SPACECOM officials

have disclosed this deviation and its impact in its amended

OPEVAL report.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 53 GAO/AIMD-00-30 Space Command Y2K Operational Testing

40

Findings: Execution

Criteria: Mission-critical “thin line” systems should be executed

using normal operating procedures, and a seamless continuity of

operations during critical Y2K date rollovers should be observed.

Finding: For the Space Control OPEVAL, 29 systems were

tested and 3 experienced “hard” failures. According to Space

Command officials, it was not necessary to invoke contingency

plans for these failures because operators were able to perform

workarounds to complete mission tasks. For 2 of the 3 “hard”

failures, these workarounds were included in OPEVAL

documentation; however, for the third “hard” failure, OPEVAL

documentation was not prepared. According to Space

Command officials, in all 3 cases, operations were not disrupted

and tasks were completed seamlessly and continually.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 54 GAO/AIMD-00-30 Space Command Y2K Operational Testing

41

Findings: Execution

Criteria: Ensure that all data needed to conduct the evaluation

for the Y2K case has been captured prior to resetting the system

to current day operations requirements.

Finding: Prior to resetting the systems to present day operational

conditions, Space Command determined that (1) the 29 “thinline” systems were assessed, (2) the master scenario events

were performed and deviations were identified, and (3) all data

needed to make an assessment of Space Command’s ability to

perform the defined “thin line” were collected and archived.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 55 GAO/AIMD-00-30 Space Command Y2K Operational Testing

42

Analysis

Defense Test Criteria Result

Categorize, document, and report failures Satisfied

Determine mission impact of Y2K failures Satisfied

Ensure Y2K OPEVAL exit criteria are met Partially Satisfied
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 56 GAO/AIMD-00-30 Space Command Y2K Operational Testing

43

Findings: Analysis

Criteria: All failures are to be identified and properly categorized

as either “hard” or “soft” failures and should be documented and

reported in accordance with the data collection and analysis

plan.

Finding: Space Command identified 5 Y2K failures during the

Space Control OPEVAL and categorized 3 as “hard” and 2 as

“soft” failures. All system failures were documented in

accordance with DOD Y2K requirements and reported to the

Joint Staff Y2K office. Examples of the “hard” failures include a

system that did not display messages during and after the leap

year rollover and a system that did not automatically list file

names for operators in a viewer window.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 57 GAO/AIMD-00-30 Space Command Y2K Operational Testing

44

Findings: Analysis

Criteria: Determine the impact of a failure on the accomplishment

of a critical mission.

Finding: Space Command determined that all 5 Y2K failures had

no significant impact on its ability to perform the Space Control

mission.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 58 GAO/AIMD-00-30 Space Command Y2K Operational Testing

45

Findings: Analysis

Criteria: JCS defined 9 exit criteria that OPEVAL results should

be measured to ensure that critical tasks and missions can be

performed in a Y2K environment.

Finding 1: Space Command measured its OPEVAL performance

against the 9 Joint Staff exit criteria and concluded that the

Space Control mission can be successfully performed in a Y2K

environment.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 59 GAO/AIMD-00-30 Space Command Y2K Operational Testing

46

Findings: Analysis

Finding 2: Space Command did not document that all the measures

of performance established as exit criteria for intelligence critical

tasks/systems were achieved. Measures of performance are used

to determine whether specified system functions are performed

within established time frames. Space Command’s measures of

performance for the critical tasks in its Space Control OPEVAL

included predetermined numbers of transactions to be executed

and time constraints within which transactions are to be executed.

However, Space Command only documented a portion of the

predetermined number of transactions specified in the OPEVAL

Plan for intelligence critical tasks. According to intelligence officials,

all predefined transactions were executed successfully but only

one-fifth were documented because it was too time-consuming to

print screens during testing. In the absence of the requisite test

results documentation, we could not validate this claim.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 60 GAO/AIMD-00-30 Space Command Y2K Operational Testing

47

Findings: Analysis

Finding 3: At the time of our briefing, Space Command had not

addressed this concern. We therefore recommended that the

final report be revised to reflect this deviation and to describe the

actions being taken to mitigate the resulting risks. Space

Command officials have since agreed that some alternative

measure should have been taken to document all test results

and thus ensure the OPEVAL’s integrity was not compromised.

The officials have also agreed to our recommendation and plan

to revise the final OPEVAL report to reflect this deviation and to

ensure that the November 1999 operational evaluation of its

intelligence mission area provides for fully documenting test

results.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 61 GAO/AIMD-00-30 Space Command Y2K Operational Testing

48

Reporting

Defense Test Criteria Result

Prepare Y2K reports describing mission impact and readiness Partially Satisfied

Provide reports to Joint Staff J7 within required time frames Satisfied
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 62 GAO/AIMD-00-30 Space Command Y2K Operational Testing

49

Findings: Reporting

Criteria: CINCs are to prepare Y2K reports describing mission

impact and readiness.

Finding 1: Space Command provided Y2K reports to the Joint

Staff which concluded that all critical tasks supporting Space

Control can be performed with no significant impact on

readiness caused by potential Y2K failures.

Finding 2: Space Command officials stated that the reports were

completed as required. However, the reports did not fully

describe the limitations in the scope of the OPEVAL and testing

deviations (i.e., the omission of 6 key communications systems,

the noncompliant system involved in the OPEVAL, configuration

changes made to a system after the baseline was established

for the OPEVAL, and the failure to fully document that
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 63 GAO/AIMD-00-30 Space Command Y2K Operational Testing

50

Findings: Reporting

established performance criteria for intelligence tasks/systems

were satisfied) or the extent to which these limitations and

deviations affected the command’s ability to draw unqualified

conclusions about Space Control mission readiness.

Finding 3: Since completing the Space Control OPEVAL, Space

Command provided documentation that showed the 6

communications systems were included in other OPEVAL or

end-to-end tests. Also, Space Command has recently agreed to

revise its 30-day report to disclose testing deviations involving

the use of a noncompliant system in the OPEVAL, the failure to

follow configuration management procedures, and the failure to

fully document intelligence tasks/systems test results.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 64 GAO/AIMD-00-30 Space Command Y2K Operational Testing

51

Findings: Reporting

Criteria: A preliminary report is required within 7 calendar days

after the completion of the OPEVAL and a final report is required

within 30 calendar days. Both reports are to be provided to Joint

Staff.

Finding: Space Command completed the 7- and 30-day reports

for the Space Control OPEVAL and provided them to Joint Staff.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 65 GAO/AIMD-00-30 Space Command Y2K Operational Testing

52

Conclusions

• Space Command satisfied many of the Defense OPEVAL

requirements for its defined Space Control “thin line.”

• However, key steps that are vital to (1) ensuring that only compliant

systems or system contingency plans are used in the OPEVAL,

(2) fully disclosing deviations from planned performance measures

and the impact of doing so, and (3) accurately reporting mission

readiness in light of OPEVAL scope limitations were not fully

satisfied. As a result, the Y2K readiness of Space Control critical

tasks involving intelligence and communications systems was not

known with sufficient surety to support Space Command’s March

1999 unqualified conclusion of mission readiness in a Y2K

environment.
Appendix I

Briefing on Results of GAO Review of 

SPACECOM Space Control Y2K OPEVAL

Page 66 GAO/AIMD-00-30 Space Command Y2K Operational Testing

53

Conclusions

• Since then, Space Command has taken steps to fill voids in its

understanding of Space Control mission readiness by ensuring that

omitted communications systems were included in other Y2K endto-end testing or OPEVAL events. It has also taken additional

action to improve testing procedures and documentation

requirements and has agreed to address our recommendation for

revising its final OPEVAL report to reflect deviations with regard to

the performance and verification of intelligence systems’ Y2K

compliance. Moreover, Space Command has acted to ensure that

its planned November 1999 operational evaluation provides for fully

documenting test results. Therefore, we are not making any further

recommendations at this time.
Page 67 GAO/AIMD-00-30 Space Command Y2K Operational Testing

Appendix II

Objectives, Scope, and Methodology ApendxIi

At the request of the Chairman, House Appropriations Committee, 

Subcommittee on Defense, we selected the Space Command Space Control 

evaluation for review to determine (1) if the evaluation was planned, 

executed, and documented in accordance with DOD guidelines, and

(2) what the evaluation results indicated concerning readiness and risks. 

This operational evaluation was selected in collaboration with the Defense 

Inspector General to ensure appropriate coverage of all combatant 

command operational evaluations and no duplication of effort.

To satisfy our first objective, we reviewed the evaluation plan, testing 

documentation and records, and test results and associated reports. We 

also interviewed Space Command officials responsible for Year 2000 

operational evaluation planning, execution, and reporting tasks. Further, 

we examined the century date rollover testing documents for the 

operational evaluation and compared Space Command’s operational 

evaluation planning, execution, analysis, and reporting actions against JCS 

operational evaluation guidance and our Year 2000 testing guide.

To satisfy the second objective, we reviewed Space Command’s operational 

evaluation results, including its 7- and 30-day reports and system problem 

tracking reports. We also interviewed Space Command officials and 

analysts responsible for developing operational evaluation assessment 

methodologies, interpreting evaluation metrics, and ensuring that 

evaluation exit criteria were met.

On October 1, 1999, we briefed Space Command leadership on the results 

of our review. Space Command provided oral comments on our briefing 

slides, and we have incorporated them as appropriate. We performed our 

work from March through October 1999 in accordance with generally 

accepted government auditing standards.
Page 68 GAO/AIMD-00-30 Space Command Y2K Operational Testing

Appendix III

GAO Contact and Staff Acknowledgements ApendxIi

GAO Contact Randolph C. Hite, (202) 512-6240

Acknowledgements In addition to those named above, Ronald B. Bageant, Cristina T. Chaplain, 

Madhav Panwar, and Yvonne Vigil made key contributions to this report.

(511661) Letr
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the 

following address, accompanied by a check or money order made 

out to the Superintendent of Documents, when necessary, VISA and 

MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are 

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and 

testimony. To receive facsimile copies of the daily list or any list 

from the past 30 days, please call (202) 512-6000 using a touchtone 

phone. A recorded menu will provide information on how to obtain 

these lists.

For information on how to access GAO reports on the INTERNET, 

send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at: 

http://www.gao.gov

United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. GI00