United States General Accounting Office

GAO Report to Congressional Committees

April 1997 TACTICAL

INTELLIGENCE

Joint STARS Full-Rate

Production Decision

Was Premature and

Risky

GAO/NSIAD-97-68

GAO United States

General Accounting Office

Washington, D.C. 20548

National Security and

International Affairs Division

B-275858 

April 25, 1997

Congressional Committees

We have reviewed the Department of Defense’s (DOD) recent decision to

commit to the full-rate production of the Joint Surveillance Target Attack

Radar System (Joint STARS). More specifically, we analyzed whether (1) the

system had demonstrated a level of maturity through testing to justify a

full-rate production commitment, (2) DOD considered and resolved

important cost and performance issues prior to making its decision, and

(3) there are future actions that could reduce program risk. This review

was performed under our basic legislative responsibility and we are

addressing it to you because it falls under your committees’ jurisdiction.

Background Joint STARS is a joint Air Force and Army wide-area surveillance and target

attack radar system designed to detect, track, classify, and support the

attack of moving and stationary ground targets. This $11 billion major

defense acquisition program consists of air and ground segments—

refurbished 707 aircraft (designated the E-8) equipped with radar,

operation and control, data processing, and communications subsystems,

together with ground stations equipped with communications and data

processing subsystems.

Low-rate initial production (LRIP)1

 of the Joint STARS aircraft began in fiscal

year 1993. In line with 10 U.S.C. 2399, DOD’s final decision to proceed

beyond LRIP first required the DOD Director of Operational Test and

Evaluation (DOT&E) to submit a report to Congress, referred to as the

Beyond LRIP report, stating whether (1) the test and evaluation performed

was adequate and (2) testing demonstrated that the system is effective and

suitable for combat, that is, operationally effective and suitable.2

The Joint STARS aircraft was scheduled to begin its initial operational test

and evaluation—referred to as the Joint STARS multi-service operational

1

Low-rate initial production of systems is to produce the minimum quantity necessary to (1) provide

production-configured or representative articles for operational test and evaluation, (2) establish an

initial production base for the system, and (3) permit an orderly increase in the production rate for the

system sufficient to lead to full-rate production upon the successful completion of operational test and

evaluation.

2

Operational effectiveness refers to the ability of a system to accomplish its mission in the planned

operational environment. Operational suitability is the degree to which a system can be placed

satisfactorily in field use considering such factors as reliability and maintainability.

Page 1 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

test and evaluation3—in November 1995. That testing was delayed and

then changed because of the deployment of Joint STARS assets to the

European theater to support Operation Joint Endeavor in Bosnia. The Air

Force Operational Test and Evaluation Center (AFOTEC) and the U.S. Army

Operational Test and Evaluation Command conducted a combined

development and operational test of Joint STARS from July through

September 1995 and an operational evaluation of the system during

Operation Joint Endeavor from January through March 1996.

Two Air Force Joint STARS aircraft and 13 Army Joint STARS ground station

modules were deployed to support Operation Joint Endeavor and

operationally evaluated from January through March 1996. After analyzing

the data from the combined development and operational test and the

operational evaluation performed during Operation Joint Endeavor,

AFOTEC issued its Joint STARS multi-service operational test and evaluation

final report on June 14, 1996. DOT&E staff analyzed the same and additional

data and the Director issued his Beyond LRIP report to Congress on

September 20, 1996.4

 On September 25, 1996, the Under Secretary of

Defense for Acquisition and Technology signed an acquisition decision

memorandum approving the Joint STARS program’s entry into full-rate

production with a total planned quantity of 19 aircraft.

Results in Brief The Joint Surveillance Target Attack Radar System’s performance during

its combined development and operational test and the operational

evaluation done in Bosnia do not support a decision to commit the system

to full-rate production. The system’s operational effectiveness and

suitability were not demonstrated during the operational testing. For

example, the DOD Director of Operational Test and Evaluation could only

state that the system had demonstrated effectiveness for “operations other

than war” and found that the system “as tested is unsuitable.” He further

reported that only 18 (25 percent) of 71 performance criteria tested were

demonstrated met by the system and that further testing is required for the

remaining 53.

3

The multi-service operational test and evaluation was to consist of a combined development and

operational test and a dedicated operational test.

4

Our analysis focused principally on the details in the “Beyond LRIP” report rather than just the

conclusions in the letter transmitting that report to Congress. Those details provide a clearer picture of

Joint STARS’ performance. For example, while the letter states “as far as suitability is concerned, [the

system] did not meet its requirements in [Operation Joint Endeavor], a problem which would be

exacerbated in a higher intensity conflict,” the report states that Joint STARS “as tested is unsuitable”

and provides detailed examples of its suitability problems.

Page 2 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

DOD’s decision to move Joint STARS into full-rate production was premature

and raised the program’s level of risk. The program could have continued

under LRIP until operational effectiveness and suitability for combat were

demonstrated and plans to address identified deficiencies and reduce

program costs were completed. Instead, DOD decided in favor of Joint

STARS full-rate production without the benefit of that information. During

the period that the full-rate production decision was being considered, the

Assistant to the President for National Security was promoting the sale of

the system to the North Atlantic Treaty Organization (NATO). In an

August 10, 1996, memorandum to the Secretaries of State, Defense, and

Commerce and to the Chairman of the Joint Chiefs of Staff, the Assistant

to the President for National Security Affairs stated that: “We have been

working through various military, diplomatic, and political channels to

secure NATO support for a Fall 1996 decision in principle by the Conference

of Armament Directors. . .to designate [Joint STARS] as NATO’s common

system.” A DOD official informed us that in November 1996, the NATO

armament directors delayed their decision on Joint STARS for a year.

Before DOD approved the full-rate production of Joint STARS, DOT&E

provided Congress with a Joint STARS Beyond LRIP report, as required by

law. The report clearly indicates that (1) further operational testing is

needed, (2) DOT&E could only declare effectiveness for operations other

than war, and (3) the system was unsuitable as tested. Having issued this

report, DOT&E is under no further obligation to report to Congress at the

Beyond LRIP report level of detail on the adequacy of the operational

testing or on whether the system has demonstrated effectiveness and

suitability for combat. However, DOD plans follow-on test and evaluation to

address the deficiencies identified during the earlier testing.

There is an opportunity not currently under consideration that could

reduce the Joint STARS’s program cost and result in an improved system.

Since the Joint STARS was approved for LRIP, the procurement cost

objective of the Air Force’s share of the Joint STARS has increased by about

$1 billion. This is primarily due to the fact that it is taking greater effort

and more resources to refurbish the 25-30 year old 707 airframes than

previously anticipated. The estimated cost of procuring, refurbishing, and

modifying each used 707 airframe to receive the system’s electronics is

now about $110 million. As early as 1992, the Boeing Company proposed

putting the system on newer Boeing 767-200 Extended Range aircraft, but

this proposal was not accepted at that time as cost-effective. Given the

current 707 airframe procurement, refurbishment, and modification cost

and a 1996 price for a commercial version Boeing 767-200 Extended Range

Page 3 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

aircraft of between $82 million and $93 million, it may now be more

cost-effective for the Air Force to buy that or some other new, more

capable aircraft. Such an aircraft could provide a longer life, greater room

for growth, greater flight range, greater fuel efficiency, higher operational

availability, and lower program life-cycle costs.

Test Results Do Not

Support Full-Rate

Production

LRIP of the Joint STARS aircraft began in fiscal year 1993. By statute, 10

U.S.C. 2399, the “Secretary of Defense shall provide that a major defense

acquisition program may not proceed beyond low-rate initial production

until initial operational test and evaluation of the program is completed.”

Operational test and evaluation is the primary means of assessing weapon

system performance in a combat-representative environment. It is defined

as the (1) field test, conducted under realistic combat conditions, to

determine an item’s effectiveness and suitability for use in combat by

typical military users and (2) evaluation of the results of such a test. If

used effectively, operational test and evaluation is a key internal control

measure to ensure that decisionmakers have objective information

available on a weapon system’s performance, thereby minimizing risks of

procuring costly and ineffective systems.

Joint STARS was moved from low-rate to full-rate production even though

(1) it performed poorly during both the combined development and

operational test and the operational evaluation in Bosnia, (2) excessive

contractor effort was needed to support Operation Joint Endeavor, (3) the

suitability and sustainability of the system is questionable since it uses

refurbished 25-30 year old airframes, and (4) operational software is

considered significantly immature.

Test Results Were

Reported as Disappointing

In DOT&E’s Beyond LRIP report, the DOT&E stated that Joint STARS had only

demonstrated effectiveness for operations other than war. The report

indicated that of three critical operational issues5

 to judge effectiveness,

only one had been demonstrated as met “. . . with limitations.” Those

critical operational issues related to (1) performance of the tactical

battlefield surveillance mission, that is, surveillance—“met with

5

DOD regulation 5000.2-R states that “critical operational issues are the operational effectiveness and

operational suitability issues (not parameters, objectives or thresholds) that must be examined in

operational test and evaluation to evaluate/assess the system’s capability to perform its mission.” It

also states that “if every critical operational issue is resolved favorably, the system should be

operationally effective and operationally suitable when employed in its intended environment by

typical users.”

Page 4 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

limitations”; (2) support of the execution of attacks against detected

targets, that is, target attack support; and (3) the provision of information

to support battlefield management and target selection, that is, battle

management. The effectiveness critical operational issues were judged

based on seven supporting measures. In its report to Congress, DOT&E

listed four of those measures of effectiveness as “not met” during the

system’s combined development and operational test and did not list any

as having been demonstrated during the Operation Joint Endeavor

operational evaluation.

However, of greater concern, according to DOT&E, is the fact that the

system did not meet its overall suitability requirements during Operation

Joint Endeavor. In his executive summary, the Director stated that most of

DOT&E’s Joint STARS concerns relate to operational suitability. He went on

to say that

“In the current configuration, the [Joint STARS] aircraft has not demonstrated the ability to

operate at the required maximum altitude; adequate tactics, techniques, or procedures to

integrate [Joint STARS] into operational theaters have not been developed; [Joint STARS]

exceeded the break rate and failed the mission reliability rate during [Operation Joint

Endeavor]. During [Operation Joint Endeavor], [Joint STARS] did not achieve the effective

time-on-station requirement.”

He concluded that without corrective actions, “[Joint STARS] would not be

suitable in higher intensity conflict” and later in the report judged that the

system “as tested is unsuitable.”

Analysis of DOT&E’s Beyond LRIP report indicates that not only did Joint

STARS have disappointing test results but also that extensive follow-on

operational testing of Joint STARS is needed. In its Beyond LRIP report,

DOT&E presented a table that reported its findings of the combined

development and operational test and Joint STARS Operation Joint

Endeavor operational evaluation and indicates where further testing is

required. Our analysis of that table indicates that at most only 25 of 71 test

criteria could be judged met. DOT&E considers 18 of those 25 to require no

further testing, that is, DOT&E judges them clearly met. However, our

analysis also indicates that 19 test criteria were clearly not met and that as

many as 26 might not have been met. Twenty-seven of the criteria could

not be determined in either the combined development and operational

test or the Operational Joint Endeavor operational evaluation. Of the 71

Joint STARS operational test and evaluation criteria listed, DOT&E indicates

that 53, or about 75 percent, require further testing.

Page 5 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

In addition to the above, DOT&E also noted that there were several

operational features present during Joint STARS Operation Joint Endeavor

deployment that were essential to its mission accomplishment but were

not included in the recent production decision. It provided two specific

examples—satellite communications and a deployable ground support

station. DOT&E believes these features “will be a necessary part of the

production decision to achieve a capable [Joint STARS] system.” It also

noted the need for other features—moving target indicator clutter

suppression, communications improvements, terrain masking tools for

ground station module operators, and linkage to operational theater

intelligence networks. Since at least two of the features present during

Operation Joint Endeavor were “essential” to its mission accomplishment

have already been developed, and may be needed “to achieve a capable

Joint STARS system,” those features should also be tested during the

planned Joint STARS follow-on test and evaluation.

Significant Contractor

Involvement

The degree of contractor involvement required during the operational

evaluation indicates increased program risk and makes the reported Joint

STARS performance appear better than it would have otherwise. The

multi-service operational test and evaluation plan, in discussing contractor

involvement during the testing, stated:

“[Multi-service operational test and evaluation] must yield the most credible and objective

results possible. All facets of the test effort must operate under the rules that support total

objectivity and prevents improper data manipulation.”

The test plan also states that interim contractor support “will be limited to

perform ground maintenance only; no in-flight support.” Regarding the

Army’s ground station modules, it states that “the Army maintenance

concept does not call for [contractor involvement] at any level . . . .”

However, during Operation Joint Endeavor there was significant

contractor support of the 2 aircraft and 13 ground station modules

deployed. According to the AFOTEC report,

“Approximately 80 contractors were deployed to support the E-8C. However, three or four

[contractor] systems engineers flew on each flight to ensure they could provide system

stability and troubleshooting expertise during missions. Additionally, three or four

[contractor] software engineers were on the ground full time, researching and developing

fixes to software problems identified during the deployment.” 

Page 6 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

AFOTEC also reported that “Each of the [ground station modules] had one

contractor representative on site and on call with additional help available

as necessary. Five contractor representatives remained at [Rhein-Main Air

Base] and functioned as a depot.” The AFOTEC report stated that the “test

director agreed to contractor participation in the [Operational Evaluation]

to a greater extent [than] permitted under US Public Law, Title 10, Section

2399.” [Emphasis added.]

When we formally expressed our concerns about the significant contractor

involvement in Operation Joint Endeavor,6 DOD did not directly

acknowledge that contractors were utilized beyond the constraints of the

law governing operational test and evaluations. It stated that “were this

solely an [initial operational test and evaluation], contractors would not

have been utilized beyond the constraints of 10 U.S.C. §2399,” and noted

that the contractors were involved in the Joint STARS operation to support

the mission. It further stated that employing Joint STARS in Operation Joint

Endeavor “allowed the system to be operated and tested at a greater

operational tempo than the system would have undergone in traditional

testing.” DOD also stated that “because of the developmental nature of the

aircraft, we needed to have more contractor personnel involved than we

would otherwise have had.”

It is understandable that DOD wanted to provide the best support possible

in Operation Joint Endeavor. However, such significant contractor use

neither supports a conclusion that the system is operationally effective or

suitable for combat, nor is it indicative of a level of system maturity that

justifies full-rate production.

Joint STARS failure to meet its maintainability criteria during an operation

less demanding than combat, even with such significant contractor

involvement beyond that planned for in combat, also raises the question of

the Air Force’s ability to develop a cost-effective maintenance plan for the

system. This issue is recognized in the Under Secretary’s acquisition

decision memorandum approving Joint STARS full-rate production. In that

memorandum, the Under Secretary called for the Air Force to fully

examine Joint STARS affordability, sustainability, and life-cycle costs,

including the scope of contractor support.

6

Joint STARS Production Decision (GAO/NSIAD-96-242R, Sept. 20, 1996).

Page 7 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

System Sustainability and

Suitability Is Questionable

In discussing the sustainability of the Joint STARS system, DOT&E noted in its

Beyond LRIP report that “It is not yet known what the operational tempo

will be for Joint STARS.” It concluded that

“If it is determined that the system will be operated at rates similar to AWACS [Airborne

Warning And Control System], it is questionable whether the [Joint STARS aircraft] can be

sustained over time. Airframe problems have already been experienced on the existing

[Joint STARS airframes], including a hydraulics failure and a cracked strut in the fuselage

between the wings.”

In discussing the Joint STARS aircraft engines, DOT&E noted that they “are

1950s technology and may not be reliable” and cited AFOTEC’s reporting

that engine failures were among the principal reasons that the aircraft

failed to meet the break rate criteria7

 during Operation Joint Endeavor.

In discussing Joint STARS suitability, DOT&E also noted that the limited

power of the engines “made it difficult to reach the aircraft’s normal

operating altitude of 36,000 feet, much less the 42,000 feet maximum

altitude it is required to reach.” It further reported that during Operation

Joint Endeavor, the aircraft required approximately 11,000 feet of runway

when taking off with 140,000 pounds of fuel and concluded that “this may

pose a significant challenge to operational commanders because the

[North Atlantic Treaty Organization] standard runway length is 8,000 feet.”

It noted that operational challenges would be great in other theaters as

well and cited Korea as an example. It reported that Joint STARS

“. . . would face operational challenges taking off from five runways in Korea, each

approximately 9,000 feet long. Operations out of Korea would likely require taking off with

less fuel and subsequent aerial refueling or shortening the time on station.”

Another area of Joint STARS suitability concern is the system’s growth

potential. DOT&E has reported that it is not clear that the remanufactured

707 platforms will be capable of incorporating all of the planned upgrades,

noting that the airframe limits the system’s growth potential both in weight

and volume. It reported that as the current mission equipment already fills

much of the fuselage, there is little room for expansion. DOT&E also noted

that increasing the payload weight would require longer takeoff runways

or taking off with less fuel, thus increasing the aerial refueling requirement

or decreasing mission duration.

7

Break rate is one measure of system suitability. For Joint STARS operational testing it was defined as

the percentage of missions flown for a specific period of time in which a previously mission-capable

essential subsystem was inoperable once the aircraft landed.

Page 8 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

DOT&E also noted that the system’s current computers limited its growth

potential due to their having very little reserve processor time or memory.

It stated that the Air Force requires that no more than 50 percent of central

processor unit cycles or memory be utilized by a new system. DOT&E

reported that “None of the E-8C computer subsystems meet these

requirements.” It provided an example of the problem, stating that “the

memory reserve of the operator workstations still does not meet the

requirement, even after being increased from 128 megabytes to 

512 megabytes just prior to [Operation Joint Endeavor].” This assessment

is another indicator of the program’s elevated risk. As DOT&E noted “Future

software enhancements and modifications may require significant

hardware upgrades. . . .”

Operational Software

Rated Significantly

Immature

The AFOTEC report specifically pointed to the lack of maturity in Joint STARS

software. For example, AFOTEC reported that

• “during Joint STARS [multi-service operational test and evaluation],

software deficiencies were noted on every E-8C subsystem;”

• the software “does not adequately support [the] operator in executing the

mission;” and

• “Joint STARS software does not show the expected maturity trends of a

system at the end of development.”

In discussing Joint STARS software maturity, DOD advised us that the AFOTEC

report judged the system overall operationally effective and suitable.

Specifically, in reference to software problems, DOD stated that “the

majority of software faults that occurred during Operation Joint Endeavor

were resolved while airborne in less than 10 minutes.” However, both

AFOTEC and DOT&E had some critical concerns regarding how Joint STARS

software functioned. For example, according to AFOTEC, the “Joint STARS

software is immature and significantly impedes the system’s reliability and

effectiveness,” and according to DOT&E

• “Immature software was clearly a problem during [Operation Joint

Endeavor]. . .”

• “. . .the prime contractor had to be called in to assist and correct 69

software-specific problems during the 41 E-8C missions . . . .an average of

1.4 critical failures per flight. . .”

• “Communications control was lost on 69 percent of the flights.”

• “The system management and control processor failed and had to be

manually reset on half of the flights.”

Page 9 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

DOD has stated that the Air Force “plans several actions to mature the

software and provide the required support resources” and that “an interim

software release in April 1997 will correct some software deficiencies

identified during the operational evaluation.” DOD also noted that software

updates will be loaded each year thereafter and that software changes are

easily incorporated. How easily these software changes are incorporated

remains to be seen because much of this software, according to AFOTEC

and DOT&E, is poorly documented. For example, AFOTEC has reported that

there are 395 deficiency reports open against the Joint STARS program, 318

of which are software related. DOT&E also stated that the more than 750,000

lines of Joint STARS software code are “poorly documented” and later

commented that “Software problems with the communications and

navigation systems were never fully corrected, even after extensive efforts

by the system contractor.”8

 These facts in combination with DOD’s

comments raise the serious question as to which software deficiencies are

to be addressed in the planned April software update.

Alternative Aircraft

Should Be Considered

There is an opportunity not currently under consideration that could

reduce the Joint STARS program cost and result in an improved system.

Since the Joint STARS was approved for LRIP, the procurement cost

objective of the Air Force’s share of the Joint STARS has increased by about

$1 billion. Program costs escalated from approximately $5.2 billion to

approximately $6.2 billion in then-year dollars. A DOD official informed us

that of the $1 billion cost growth, $760 million is attributed to the

increased cost to buy, refurbish, and modify the used 707 airframes to

receive the Joint STARS electronics. The remaining cost growth is attributed

to other support requirements and growth in required spare parts.

At least as early as 1992, the Boeing Company proposed putting Joint STARS

on newer Boeing 767-200 Extended Range aircraft, but this proposal was

not accepted as cost-effective. According to the 1996 Boeing price list, the

commercial version of this aircraft can be bought for between $82 million

and $93 million depending on options chosen (this is flyaway cost—the

cost of a plane ready to be flown in its intended use). Furthermore, the

flyaway cost of a commercial Boeing 757, which a Boeing representative

informed us is in many respects more comparable to the 707s being used,

is listed at between $61 million to $68 million. The actual cost of procuring

either of these aircraft could be lowered by volume discounts and by the

8

AFOTEC stated that “Safety of flight is jeopardized due to invalid navigational commands sent from

the FMS-800 [flight management system] and conflicting and/or invalid navigational data displayed to

the flight crew.”

Page 10 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

cost of the commercial amenities not required. On the other hand, these

aircraft would require modifications to receive Joint STARS equipment,

which would raise their cost.

DOD informed us that the cost of procuring, refurbishing, and modifying the

current 707 aircraft to receive Joint STARS equipment is now estimated to

be about $110 million per airframe. The cost of procuring and preparing

new aircraft might be comparable or even less than the current cost. In

addition, the Air Force would acquire a new platform that could have

(1) greater room for growth (both volume and weight), (2) take off

capability from a shorter runway, (3) greater time-on-station capability,

(4) significantly improved fuel efficiency, (5) extended aircraft life over

the 707 currently used, and (6) reduced operational and support cost.

In commenting on a draft of this report, DOD stated that it considered

alternatives to the current air platform, both before LRIP started and at the

full-rate production decision point. It also stated that the cost of moving

the Joint STARS mission to an alternative platform would outweigh the

benefits. We note, however, that at a meeting with DOD and service officials

to discuss that draft, we asked about the reported DOD and service

analyses. One Air Force official stated that the Air Force’s platform choice

was not revisited prior to the full-rate production decision. None of the

other 13 DOD and service officials present objected to that statement.

Furthermore, when we asked for copies of the air platform analyses that

were done in support of either the low-rate or the full-rate production

decision, DOD was unable to supply those analyses. Finally, DOD officials

have informed us that a Command, Control, Communications, Computers,

Intelligence, Surveillance, and Reconnaissance Mission Assessment9

 has

been performed that indicates that the Air Force could acquire a more

effective system while saving $3 billion through the year 2010 by moving

the Joint STARS mission to either a business jet or an unmanned aerial

vehicle following the procurement of the twelfth current version Joint

STARS aircraft.

9

The results of this mission assessment are being considered in the Quadrennial Defense Review, a

process in which DOD is taking a fundamental look at its strategy, force structure, modernization,

infrastructure, and readiness to meet future mission requirements.

Page 11 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

Production

Commitment

Unnecessary and

Risky

We have previously informed DOD of our concerns about the decision to

move to full-rate production in spite of the numerous testing deficiencies

reported by both AFOTEC and DOT&E. DOD responded that in making the

decision to move to full-rate production, it “considered the test reports

(both the services’ and the Director, Operational Test and Evaluation’s),

the plans to address the deficiencies identified during developmental and

operational testing, cost estimates, operational requirements, and other

program information.”

Although DOD believes that “none of the deficiencies identified are of a

scope that warrants interrupting production,” the production decision

memorandum clearly reflects a recognition that this program carries

significant risk. In his memorandum, the Under Secretary of Defense for

Acquisition and Technology directed (1) an update of the Joint STARS Test

and Evaluation Master Plan to “address multi-service [operational test and

evaluation] deficiencies (regression testing);” (2) acceleration of the

objective and threshold dates for the planned Follow-on Operational Test

and Evaluation; and (3) the Air Force to “fully examine [Joint STARS]

affordability, sustainability, and life cycle costs10 including the scope of

contractor use for field-level system support.”

Factors other than system performance may have influenced the decision

to move Joint STARS forward into full-rate production. DOD’s full-rate

production decision for this program occurred during the same time frame

the Joint STARS system was actively being promoted as the U.S.

government’s candidate for meeting NATO’s military requirement for a

ground surveillance system. For example, in an August 10, 1996,

memorandum to the Secretaries of State, Defense, and Commerce and to

the Chairman of the Joint Chiefs of Staff, the Assistant to the President for

National Security Affairs stated that “We have been working through

various military, diplomatic and political channels to secure NATO support

for a Fall 1996 decision in principle by the Conference of Armament

Directors . . . to designate [Joint STARS] as NATO’s common system.” He

went on to state that

“I am writing to be sure you know that the President is personally committed to [Joint

STARS], has engaged Chancellor Kohl on this issue and will continue his personal

involvement with key allies to ensure our goal is achieved. I would ask that you underscore

your personal support for our collective efforts on behalf of [Joint STARS] when you meet

with your NATO and European counterparts.”

10We believe this analysis should include examining the use of new airframes.

Page 12 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

Notwithstanding DOD’s September 1996 commitment to full-rate Joint STARS

production, a DOD official informed us that the NATO armament directors in

their November 1996 meeting delayed for 1 year any decision on

designating Joint STARS as NATO’s common system or pursuing an alternate

system to be developed.

Follow-on Operational

Test and Evaluation

Planned

In the process of moving the Joint STARS program forward into full-rate

production, DOD produced a Beyond LRIP report for Congress and thus

moved past a key congressional reporting requirement that serves as an

important risk management mechanism. The Beyond LRIP report to

Congress that is required before major defense acquisition programs can

proceed into full-rate production serves to inform Congress of the

adequacy of the operational testing done on the system and to provide it

with a determination of whether the system has demonstrated

effectiveness and suitability. Having issued this report, DOT&E is under no

further obligation to report to Congress at the Beyond LRIP report level of

detail on the adequacy of the operational testing or on whether the system

has demonstrated effectiveness and suitability for combat. However, DOD

plans follow-on test and evaluation of the system to address the

deficiencies identified during the system’s earlier testing.

On September 20, 1996, DOT&E sent to Congress a Joint STARS “Beyond LRIP”

report that (1) clearly indicates that further operational testing is needed,

(2) could only declare effectiveness for operations other than war, and

(3) stated that Joint STARS is unsuitable as tested. On September 25, 1996,

DOD approved the full-rate production of Joint STARS. In the acquisition

memorandum approving Joint STARS full-rate production, the Under

Secretary of Defense for Acquisition and Technology called for an

accelerated follow-on operational test and evaluation of Joint STARS that is

to address the deficiencies identified in the initial operational test and

evaluation DOT&E reported on in the Beyond LRIP report to Congress. The

planned follow-on operational test and evaluation will provide an

opportunity to judge the Joint STARS program’s progress in resolving the

issues identified in earlier testing.

Conclusions Notwithstanding any concurrent efforts to have Joint STARS designated as a

NATO common system, Joint STARS test performance and the clearly

unresolved questions about its operational suitability and affordability

should have, in our opinion, caused DOD to delay the full-rate production

decision until (1) the system had, through the planned follow-on

Page 13 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

operational test and evaluation, demonstrated operational effectiveness

and suitability; (2) the Air Force had completed an updated analysis of

alternatives for the Joint STARS to address the identified aircraft suitability

and cost issues; and (3) the Air Force had developed an analysis to

determine whether a cost-effective maintenance concept could be

designed for the system. Furthermore, as they were judged “essential” to

mission accomplishment and needed “to achieve a capable Joint STARS

system,” the satellite communications and deployable ground support

station features (present, but untested, during Operation Joint Endeavor)

should also be tested during the planned Joint STARS follow-on operational

test and evaluation.

Concerns of the magnitude discussed in this report are not indicative of a

system ready for full-rate production. The program should have continued

under LRIP until the issues identified by AFOTEC and DOT&E were resolved

and the system was shown to be effective and suitable for combat.

Furthermore, the recent cost growth related to refurbishing and modifying

the old airframes being used for Joint STARS and questions regarding the

suitability of those platforms indicate an opportunity to reduce the

program’s cost and improve the systems acquired. We believe, therefore,

that an updated study of the cost effectiveness of placing Joint STARS on

new, more capable aircraft is warranted.

Recommendation We recommend that the Secretary of Defense direct the Air Force to

perform an analysis of possible alternatives to the current Joint STARS air

platform, to include placing this system on a new airframe.

Matters for

Congressional

Consideration

Because of (1) DOD’s decision to commit to full-rate production in the face

of the test results discussed in this report and (2) its subsequent decision

to do additional tests while in production to address previous test

deficiencies, we are convinced that DOD plans to proceed with the

program. However, if Congress agrees that there is unnecessarily high risk

in this program and believes the risk should be reduced, it may wish to

require that:

• The Air Force obtain DOT&E approval of a revised test and evaluation

master plan (and all plans for the tests called for in that master plan) for

follow-on operational testing to include adequate coverage of gaps left by

prior testing and include testing of any added features considered part of

Page 14 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

the standard production configuration and that DOT&E considers key

system components.

• DOT&E provide a follow-on test and evaluation report to Congress

evaluating the adequacy of all testing performed to judge operational

effectiveness and suitability for combat and a definitive statement stating

whether the system has demonstrated operational effectiveness and

suitability.

• DOD develop and provide Congress an analysis of alternatives report on the

Joint STARS air platform that considers the suitability of the current

platform and other cost-effective alternatives, and the life-cycle costs of

the current platform and best alternatives.

Agency Comments

and Our Evaluation

In commenting on a draft of this report, DOD disagreed with our

recommendation that the Air Force be directed to perform an analysis of

possible alternatives to the current Joint STARS air platform. It also

disagreed with our suggestion that Congress may wish to require DOD to

develop and provide Congress a report on that analysis. DOD stated that

alternative platforms were considered prior to both the start of LRIP and

the full-rate production decision. DOD stated that based on (1) the fact that

over half the fleet is already in the remanufacturing process or delivered to

the user; (2) the large nonrecurring costs that would be associated with

moving the Joint STARS mission to a different platform; (3) the additional

cost to operate and maintain a split fleet of Joint STARS airframes; and

(4) the expected 4-year gap in deliveries, such a strategy would force the

costs of moving the Joint STARS mission to a different platform outweigh

the benefits.

DOD’s comment about having previously considered alternative platforms

is inconsistent with the information we developed during our review and

with Air Force comments provided at our exit conference. In an effort to

reconcile this inconsistency, we requested copies of the prior analyses of

alternative platforms, but DOD was not able to provide them. DOD’s

statement that the costs of moving the Joint STARS mission to another

platform would outweigh the benefits contradicts Command, Control,

Communications, Computers, Intelligence, Surveillance, and

Reconnaissance Mission Assessment briefings given the Quadrennial

Defense Review. Those briefings recommend (1) limiting Joint STARS

production to 12 aircraft, (2) moving the Joint STARS mission to either

corporate jets or Unmanned Aerial Vehicles, and (3) phasing out Joint

STARS 707 variants as quickly as the new platform acquisitions will allow.

According to those briefings, implementation of this recommendation

Page 15 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

could result in a more effective system and save over $3 billion through

fiscal year 2010. We believe that the issue clearly warrants further

consideration. Furthermore, given DOD’s resistance to the concept, we are

more convinced of the merits of our suggestion that Congress might wish

to require a report on such an analysis.

In commenting on our draft report, DOD also indicated that congressional

direction was unneeded on our suggestions that Congress might wish to

require (1) DOT&E approval of a revised test and evaluation master plan for

the planned Joint STARS follow-on operational test and evaluation and

(2) DOT&E to provide Congress with a follow-on operational test and

evaluation report on the adequacy of Joint STARS testing and stating

whether Joint STARS has demonstrated operational effectiveness and

suitability. DOD stated that congressional direction on the first point was

unneeded because the Joint STARS full-rate production decision

memorandum required that the test and evaluation master plan be updated

for Office of the Secretary of Defense approval and current DOD policy is

that DOT&E will review, approve, and report on oversight systems in

follow-on operational test and evaluation. DOD also stated that

congressional direction on the second point is unneeded because DOT&E

has retained Joint STARS on its list of programs for oversight and is to

report on the system in its annual report to Congress as appropriate.

DOD’s response did not directly address our point since, as DOD pointed out,

the acquisition decision memorandum that approved full-rate production

required Office of the Secretary of Defense approval, not DOT&E approval,

of the follow-on operational test and evaluation master plan. During the

course of our review, DOD officials informed us that there was significant

disagreement between the Air Force and DOT&E as to what follow-on

testing was needed. It was indicated that the issue would probably have to

be resolved at higher levels within the department, an indication of greater

flexibility than DOD implies. Furthermore, while DOD stated there were

some improvements and enhancements “that could benefit the warfighter”

and acknowledged that those features were not tested, it did not respond

to our comments that DOT&E judged those features “essential” to mission

accomplishment or commit to their operational test and evaluation. Given

these facts, we have not only maintained our suggestion that Congress

may wish to require the Air Force to obtain DOT&E approval of a revised

test and evaluation master plan, but also strengthened it to include DOT&E

approval of supporting test plans.

Page 16 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

In its response to our suggestion that Congress may wish to require that

DOT&E provide it a detailed, follow-on test and evaluation report, DOD states

congressional direction is unnecessary as DOT&E will report on the system,

among many others, in its annual report to Congress. DOD’s comment fails

to recognize, however, that we are suggesting that, given the already

reported test results, Congress may wish a more detailed report outlining

the adequacy of and the system’s performance during follow-on

operational testing to help in its oversight and provide it assurance that

the system’s problems have been substantially resolved. Given that

(1) Congress felt such reporting to be beneficial enough to require it

before a system can proceed beyond LRIP and (2) the fact that DOT&E, in the

required report provided for Joint STARS, could not certify effectiveness for

war and found the system unsuitable as tested, we continue to believe that

Congress may wish to require a similar report based on the follow-on

operational test and evaluation planned.

DOD’s comments are reprinted in their entirety in appendix I, along with

our evaluation.

Scope and

Methodology

To determine whether Joint STARS test performance indicates a maturity

justifying full-rate production, we interviewed officials and reviewed

documents in Washington, D.C., from the DOD Office of the Director of

Operational Test and Evaluation and the Joint STARS Integrated Product

Team. We reviewed the Air Force Operational Test and Evaluation

Center’s multi-service operational test and evaluation plan and its final

report on that testing and the DOD Director of Operational Test and

Evaluation’s Beyond LRIP report. To determine whether DOD considered

and resolved important cost and performance issues prior to making its

full-rate production decision, we reviewed Joint STARS program budget

documents and program-related memoranda issued by the Under

Secretary of Defense for Acquisition and Technology. To determine

whether it is possible that a more useful operational test and evaluation

report can be provided Congress, we reviewed the statute governing

operational testing and evaluation, examined DOT&E’s Beyond LRIP report,

and considered other relevant program information. We considered and

incorporated where appropriate DOD’s response to our September 20, 1996,

letter of inquiry11 and its response to a draft of this report. We conducted

this review from October 1996 through April 1997 in accordance with

generally accepted government auditing standards.

11Joint STARS Production Decision (GAO/NSIAD-96-242R, Sept. 20, 1996).

Page 17 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

We are sending copies of this letter to other appropriate congressional

committees; the Director, Office of Management and Budget; and the

Secretaries of Defense, the Army, and the Air Force. Copies will also be

made available to others upon request.

If you or your staff have any questions, please contact me, Mr. Charles F.

Rey, Assistant Director, or Mr. Bruce Thomas, Evaluator-in-Charge, at

(202) 512-4841.

Thomas J. Schulz

Associate Director,

Defense Acquisitions Issues

Page 18 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

List of Congressional Committees

The Honorable Strom Thurmond

Chairman

The Honorable Carl Levin

Ranking Minority Member

Committee on Armed Services

United States Senate

The Honorable Ted Stevens

Chairman

The Honorable Daniel K. Inouye

Ranking Minority Member

Subcommittee on Defense

Committee on Appropriations

United States Senate

The Honorable Floyd D. Spence

Chairman

The Honorable Ronald V. Dellums

Ranking Minority Member

Committee on National Security

House of Representatives

The Honorable C.W. Bill Young

Chairman

The Honorable John P. Murtha

Ranking Minority Member

Subcommittee on National Security

Committee on Appropriations

House of Representatives

The Honorable Richard C. Shelby

Chairman

The Honorable J. Robert Kerrey

Vice Chairman

Select Committee on Intelligence

United States Senate

The Honorable Porter J. Goss

Chairman

The Honorable Norman D. Dicks

Ranking Minority Member

Page 19 GAO/NSIAD-97-68 Joint STARS Production Decision
B-275858 

Select Committee on Intelligence

House of Representatives

Page 20 GAO/NSIAD-97-68 Joint STARS Production Decision
Page 21 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.

See pp. 10-11 & 15-16.

See comment 1.

See pp. 10-11 & 15-16.

Page 22 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

Now on p. 1.

Now on p. 2.

See comment 2.

Now on p. 2.

Now on p. 3.

See comment 3.

Now on p. 3.

Page 23 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

See comment 4.

Now on p. 4.

Now on p. 4.

See comment 5.

See comment 6.

Page 24 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

See comment 3.

See comment 7.

See comment 8.

See comment 8.

Page 25 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

See comment 8.

See comment 8.

See comment 4.

Now on pp. 4-6.

See comment 3.

Page 26 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

Now on p. 6.

See comment 9.

Now on pp. 6-7.

See comment 10.

Now on p. 7.

See comment 7.

Page 27 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

Now on p. 7.

See comment 11.

See comment 10.

See comment 6.

Now on p. 8.

See comment 12.

See comment 11.

See comment 12.

Page 28 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

See comment 6.

Now on p. 8.

See comment 13.

Now on p. 8.

Now on p. 9.

Now on p. 9.

See comment 14.

Page 29 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

Now on p. 10.

Now on p. 10.

See comment 8.

Now on p. 12.

Now on p. 12.

See comment 2.

Now on p. 12.

Page 30 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

See comment 15.

See comment 16.

Now on p. 14.

See comment 8.

Now on pp. 14-15.

See comments 1 and 17.

Page 31 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

Now on p. 15.

See comments 1 and 18.

Now on p. 15.

See comment 8.

Page 32 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

The following are GAO’s comments on the Department of Defense’s (DOD)

letter dated March 31, 1997.

GAO Comments 1. We have not suggested or recommended that Joint STARS production be

interrupted. We have, however, suggested actions that we believe (1) will

help reduce the program’s risk; (2) could result in the acquisition of a

more effective, less costly system; and (3) could help decisionmakers

ensure that the Joint STARS program continues to make progress.

2. The report has been modified in light of DOD’s comments.

3. DOD’s indication that other factors were considered in deciding to

proceed to full-rate production is a signal that DOD and the Air Force are

willing to accept a high level of risk even when the Director, Operation,

Test, and Evaluation (DOT&E) has concluded that the system was

unsuitable as tested and operational effectiveness for war remains to be

demonstrated. We believe, given the system’s test performance as reported

by both the Air Force Operational Test and Evaluation Command (AFOTEC)

and DOT&E and the program’s procurement cost growth of $1 billion

between the low-rate and full-rate production decision points, that an

informed full-rate production decision required the following information:

(1) an approved test and evaluation master plan for follow-on operational

testing and specific plans for the tests called for in that master plan,

(2) the results of the already ongoing study of ways to reduce the

program’s cost, and (3) an analysis of alternatives to the current platform.

DOD did not have these items in hand when it made its decision. We must

also note that DOD implies that our recommendations would require a

break in production. This is inaccurate. As we stated in the body of our

report, the program could have continued under low-rate initial

production (LRIP) until operational effectiveness and suitability for combat

were demonstrated and plans to address identified deficiencies and reduce

program costs were completed.

4. In its report on the Joint STARS multi-service operational test and

evaluation, AFOTEC stated that “Joint STARS software is immature and

significantly impedes the system’s reliability and effectiveness.” We do not

believe that, given the software intensive nature of the system, this

statement supports a conclusion that the system could be judged

operationally effective.

Page 33 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

5. We must note that follow-on operational test and evaluation of the

system was planned before the full-rate production decision. The full-rate

production decision called for acceleration of that testing and for that

testing to address deficiencies identified in the earlier tests. Joint STARS

could have continued under LRIP pending a demonstration of operational

effectiveness and suitability.

6. This speaks to the number of aircraft missions planned and the number

for which an aircraft was provided. It does not address the quality or

quantity of the support provided during those missions. Furthermore,

DOD’s comment refers to the same—operation that is reported on in both

the Air Force and DOT&E reports and in this report.

7. U.S.-based contractor support was utilized during the first Operation

Joint Endeavor deployment. It is also our understanding that during the

second Operation Joint Endeavor deployment the Air Force may have

utilized a “reach-back” maintenance concept in which U.S. stationed

contractor staff were providing field support through satellite

communications. Moreover, DOD and Air Force officials told us that at least

at the beginning of the second Operation Joint Endeavor deployment,

contractor staff were flying on the deployed aircraft. This clearly raises the

question of what the overall level of contractor support was for both the

first and second deployments.

More importantly, a decrease in the level of contractor support between

the two Operation Joint Endeavor deployments does not speak to (1) the

poor test results during the first deployment with, and in spite of, the level

of contractor support or (2) the quality of the system’s performance during

the second deployment; that is, there was no independent—DOT&E—

measurement or observation of how the system performed against its

operational requirements.

8. Given that (1) the procurement cost growth of $760 million for 19 Joint

STARS aircraft since the low-rate production decision; (2) a current 

707 airframe purchase, refurbishment, and modification cost of about

$110 million; (3) the age of the current airframes—25 to 30 years; and

(4) the $7 billion estimated operations and maintenance life-cycle cost of

those aircraft, we continue to believe that an analysis of alternatives to the

current air platform should be performed, a belief bolstered by DOD’s

inability to provide copies of its reported analyses and by the

recommendations of the Command, Control, Communications,

Computers, Intelligence, Surveillance, and Reconnaissance Mission

Page 34 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

Assessment, as discussed on pages 10, 11, 15, and 16 of this report. We

believe that the issue clearly warrants further study. Furthermore, given

DOD’s resistance to the concept, we are more convinced of the merits of

our suggestion that Congress might wish to require a report on such an

analysis.

9. DOD stated that there were some improvements and enhancements “that

could benefit the warfighter” but acknowledges that those features were

not tested. It did not respond, however, to our comments that DOT&E

judged some of those features “essential” to mission accomplishment.

Furthermore, it did not state whether those features would be subjected to

operational testing and evaluation. As we stated in the body of our report,

since at least two of the features present during Operation Joint Endeavor

were “essential” to its mission accomplishment, have already been

developed, and may be needed “to achieve a capable Joint STARS system,”

those features should also be tested during the planned Joint STARS

follow-on test and evaluation.

10. Given the level of contractor support during the multi-service

operational test and evaluation, we are unable to understand how that

support could not have impacted upon the system’s test results. As we

stated in our report, AFOTEC reported that 80 contractors were deployed,

“three or four [contractor] systems engineers flew on each flight to ensure

they could provide system stability and troubleshooting expertise during

missions,” and “three or four . . . were on the ground full time, researching

and developing fixes to software problems . . . .” Furthermore, in its report

DOT&E states that “While these [contractor] needs show that [Joint STARS]

requires sophisticated support, they may also mask certain deficiencies.” It

also reported that

“As already discussed, extensive efforts by the system contractor were required to achieve

the demonstrated availability for the E-8C aircraft. Even with those efforts the system was

not able to meet the user criteria for several measures directly related to the maintenance

concept in place during [Operation Joint Endeavor]—a concept that involved considerably

more contractor support than previously envisioned.”

11. As we noted in the body of our report, Joint STARS failed to meet test

criteria during an operation less demanding than combat, even with such

significant contractor involvement beyond that planned for in combat. In

discussing operational tempo in its Beyond LRIP report, DOT&E stated that if

the system is operated at rates similar to the Airborne Warning and

Control System, “it is questionable whether the [Joint STARS aircraft] can

Page 35 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

be sustained over time.” DOD commented that an unbiased assessment of

the measure of Joint STARS’ ability to maintain the required tempo could

not be made and would be tested during the follow-on operational test and

evaluation. We believe that an informed full-rate production decision

requires knowledge of a system’s ability to satisfy the operational tempo

expected of it. DOD made its Joint STARS full-rate production decision

without this knowledge.

12. We understand that Joint STARS, like most systems, has limitations that

need to be planned around. At issue here is a question of how great those

limitations are and whether they are acceptable. DOD states that “the user

is satisfied that the system meets requirements.” However, we must note

that the Air Force’s own Operational Test and Evaluation Center reported

that the “two critical suitability [measures of performance, sortie

generation rate and mission reliability rate], were affected by [Operation

Joint Endeavor] contingency requirements and system stability problems.”

AFOTEC stated that the sortie generation rate performance was

undetermined and judged the other critical suitability measure of

performance—mission reliability rate—as not being met. In discussing the

later critical measure of performance AFOTEC reported that

“The high failure rate of aging aircraft components affected [mission reliability rates] as

critical failures were statistically determined to affect over 30 percent of the sorties flown.

Analysis revealed the elevated critical failure rate was steady and showed no potential for

improvement. Technical data and software immaturity affected the maintainability of the

aircraft, and contractor involvement further compromised clear insight into the Air Force

technicians’ ability to repair the system.”

AFOTEC also reported on Joint STARS performance relative to 15 supporting

suitability criteria. It stated “Eight did not meet users’ criteria. One was not

tested. Only one . . . met the users’ criteria. The remaining five are reported

using narrative results.”

13. DOD discusses only the weight growth of funded activities, leaving open

the question of whether there are future, but currently unfunded,

improvements planned that will add weight growth. Air Force officials told

us that the Airborne Warning and Control System had experienced weight

growth over the life of its program. That growth was attributed to the

system’s being given added tasks over time. We believe it reasonable to

expect that the Joint STARS program experience might track that of the

Airborne Warning and Control Systems program, that is, be given added

Page 36 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

tasks and face weight growth as a result. Also, regarding Joint STARS room

for growth, DOD previously advised us that Joint STARS currently has about

455,000 cubic inches of space available. We must note that this equates to

a volume of under 7 feet cubed and that in commenting on the system’s

space limitation, DOT&E stated “There is little room available for additional

people or operator workstations.”

14. As we stated in the body of our report, how easily these software

changes are incorporated remains to be seen.

15. We requested and DOD provided additional information on this point.

DOD’s subsequent response indicates that this DOD comment was in error.

In its subsequent response, DOD stated that the follow-on test and

evaluation was accelerated “to reflect [Office of the Secretary of Defense]

desire for earlier [operational test and evaluation] to evaluate fixes to

[multi-service operational test and evaluation] deficiencies.” We believe

this statement reflects a recognition of increased program risk.

16. The acquisition decision memorandum approving Joint STARS

production clearly indicates that the Skantze study1 mentioned was not

completed at that time. We believe that the full-rate production decision

should have been made with the Skantze study in hand. Furthermore, we

do not understand why DOD felt the need to direct the Air Force to fund

and implement a plan that is to save it money, but felt no need to direct the

Air Force to examine alternative platforms that at least one other DOD

panel had stated would not only save $3 billion but also provide greater

effectiveness.

17. We believe that not only should DOT&E approval of the Joint STARS Test

and Evaluation Master Plan be required, but also of all supporting test

plans. We have changed the language of this matter for congressional

consideration accordingly.

18. We are suggesting that Congress may wish to request a more detailed

report, one at the Beyond LRIP report level of detail, a level of detail not

provided in DOT&E’s annual report. Given that DOT&E could only state

effectiveness for operations other than war—could not state a belief as to

whether the system would be effective in two of the three critical

operational roles it is expected to perform in war—and found the system

unsuitable as tested, we believe that such report would help Congress

1

This study was an affordability review of Joint STARS that examined the affordability, sustainability,

and life-cycle costs to thoroughly evaluate and control program costs.

Page 37 GAO/NSIAD-97-68 Joint STARS Production Decision
Appendix I 

Comments From the Department of Defense

maintain program oversight. DOD’s comment of “other reports as

appropriate” leaves the matter in DOD’s hand to decide if Congress would

benefit from such a report.

(702226) Page 38 GAO/NSIAD-97-68 Joint STARS Production Decision
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100