Tactical Intelligence: Joint STARS Full-Rate Production Decision Was Premature and Risky
Released: 1997-04-25T00:00:00

Letter

Background

Joint STARS is a joint Air Force and Army wide-area surveillance and target attack radar system designed to detect, track, classify, and support the attack of moving and stationary ground targets. This $11 billion major defense acquisition program consists of air and ground segments— refurbished 707 aircraft (designated the E-8) equipped with radar, operation and control, data processing, and communications subsystems, together with ground stations equipped with communications and data processing subsystems.

Low-rate initial production (LRIP) of the Joint STARS aircraft began in fiscal year 1993. In line with 10 U.S.C. 2399, DOD’s final decision to proceed beyond LRIP first required the DOD Director of Operational Test and Evaluation (DOT&E) to submit a report to Congress, referred to as the Beyond LRIP report, stating whether (1) the test and evaluation performed was adequate and (2) testing demonstrated that the system is effective and suitable for combat, that is, operationally effective and suitable.

The Joint STARS aircraft was scheduled to begin its initial operational test and evaluation—referred to as the Joint STARS multi-service operational test and evaluation—in November 1995. That testing was delayed and then changed because of the deployment of Joint STARS assets to the European theater to support Operation Joint Endeavor in Bosnia. The Air Force Operational Test and Evaluation Center (AFOTEC) and the U.S. Army Operational Test and Evaluation Command conducted a combined development and operational test of Joint STARS from July through September 1995 and an operational evaluation of the system during Operation Joint Endeavor from January through March 1996.

Two Air Force Joint STARS aircraft and 13 Army Joint STARS ground station modules were deployed to support Operation Joint Endeavor and operationally evaluated from January through March 1996. After analyzing the data from the combined development and operational test and the operational evaluation performed during Operation Joint Endeavor, AFOTEC issued its Joint STARS multi-service operational test and evaluation final report on June 14, 1996. DOT&E staff analyzed the same and additional data and the Director issued his Beyond LRIP report to Congress on September 20, 1996. On September 25, 1996, the Under Secretary of Defense for Acquisition and Technology signed an acquisition decision memorandum approving the Joint STARS program’s entry into full-rate production with a total planned quantity of 19 aircraft.

Test Results Do Not Support Full-Rate Production

LRIP of the Joint STARS aircraft began in fiscal year 1993. By statute, 10 U.S.C. 2399, the “Secretary of Defense shall provide that a major defense acquisition program may not proceed beyond low-rate initial production until initial operational test and evaluation of the program is completed.”

Operational test and evaluation is the primary means of assessing weapon system performance in a combat-representative environment. It is defined as the (1) field test, conducted under realistic combat conditions, to determine an item’s effectiveness and suitability for use in combat by typical military users and (2) evaluation of the results of such a test. If used effectively, operational test and evaluation is a key internal control measure to ensure that decisionmakers have objective information available on a weapon system’s performance, thereby minimizing risks of procuring costly and ineffective systems.

Joint STARS was moved from low-rate to full-rate production even though (1) it performed poorly during both the combined development and operational test and the operational evaluation in Bosnia, (2) excessive contractor effort was needed to support Operation Joint Endeavor, (3) the suitability and sustainability of the system is questionable since it uses refurbished 25-30 year old airframes, and (4) operational software is considered significantly immature.

Test Results Were Reported as Disappointing

In DOT&E’s Beyond LRIP report, the DOT&E stated that Joint STARS had only demonstrated effectiveness for operations other than war. The report indicated that of three critical operational issues to judge effectiveness, only one had been demonstrated as met “. . . with limitations.” Those critical operational issues related to (1) performance of the tactical battlefield surveillance mission, that is, surveillance—“met with limitations”; (2) support of the execution of attacks against detected targets, that is, target attack support; and (3) the provision of information to support battlefield management and target selection, that is, battle management. The effectiveness critical operational issues were judged based on seven supporting measures. In its report to Congress, DOT&E listed four of those measures of effectiveness as “not met” during the system’s combined development and operational test and did not list any as having been demonstrated during the Operation Joint Endeavor operational evaluation. “In the current configuration, the [Joint STARS] aircraft has not demonstrated the ability to operate at the required maximum altitude; adequate tactics, techniques, or procedures to integrate [Joint STARS] into operational theaters have not been developed; [Joint STARS] exceeded the break rate and failed the mission reliability rate during [Operation Joint Endeavor]. During , [Joint STARS] did not achieve the effective time-on-station requirement.”

He concluded that without corrective actions, “[Joint STARS] would not be suitable in higher intensity conflict” and later in the report judged that the system “as tested is unsuitable.”

Analysis of DOT&E’s Beyond LRIP report indicates that not only did Joint STARS have disappointing test results but also that extensive follow-on operational testing of Joint STARS is needed. In its Beyond LRIP report, DOT&E presented a table that reported its findings of the combined development and operational test and Joint STARS Operation Joint Endeavor operational evaluation and indicates where further testing is required. Our analysis of that table indicates that at most only 25 of 71 test criteria could be judged met. DOT&E considers 18 of those 25 to require no further testing, that is, DOT&E judges them clearly met. However, our analysis also indicates that 19 test criteria were clearly not met and that as many as 26 might not have been met. Twenty-seven of the criteria could not be determined in either the combined development and operational test or the Operational Joint Endeavor operational evaluation. Of the 71 Joint STARS operational test and evaluation criteria listed, DOT&E indicates that 53, or about 75 percent, require further testing.

In addition to the above, DOT&E also noted that there were several operational features present during Joint STARS Operation Joint Endeavor deployment that were essential to its mission accomplishment but were not included in the recent production decision. It provided two specific examples—satellite communications and a deployable ground support station. DOT&E believes these features “will be a necessary part of the production decision to achieve a capable [Joint STARS] system.” It also noted the need for other features—moving target indicator clutter suppression, communications improvements, terrain masking tools for ground station module operators, and linkage to operational theater intelligence networks. Since at least two of the features present during Operation Joint Endeavor were “essential” to its mission accomplishment have already been developed, and may be needed “to achieve a capable Joint STARS system,” those features should also be tested during the planned Joint STARS follow-on test and evaluation.

Significant Contractor Involvement

“ must yield the most credible and objective results possible. All facets of the test effort must operate under the rules that support total objectivity and prevents improper data manipulation.”

The test plan also states that interim contractor support “will be limited to perform ground maintenance only; no in-flight support.” Regarding the Army’s ground station modules, it states that “the Army maintenance concept does not call for at any level . . . .” “Approximately 80 contractors were deployed to support the E-8C. However, three or four systems engineers flew on each flight to ensure they could provide system stability and troubleshooting expertise during missions. Additionally, three or four software engineers were on the ground full time, researching and developing fixes to software problems identified during the deployment.”

AFOTEC also reported that “Each of the had one contractor representative on site and on call with additional help available as necessary. Five contractor representatives remained at [Rhein-Main Air Base] and functioned as a depot.” The AFOTEC report stated that the “test director agreed to contractor participation in the to a greater extent permitted under US Public Law, Title 10, Section 2399.”

When we formally expressed our concerns about the significant contractor involvement in Operation Joint Endeavor, DOD did not directly acknowledge that contractors were utilized beyond the constraints of the law governing operational test and evaluations. It stated that “were this solely an , contractors would not have been utilized beyond the constraints of 10 U.S.C. §2399,” and noted that the contractors were involved in the Joint STARS operation to support the mission. It further stated that employing Joint STARS in Operation Joint Endeavor “allowed the system to be operated and tested at a greater operational tempo than the system would have undergone in traditional testing.” DOD also stated that “because of the developmental nature of the aircraft, we needed to have more contractor personnel involved than we would otherwise have had.”

It is understandable that DOD wanted to provide the best support possible in Operation Joint Endeavor. However, such significant contractor use neither supports a conclusion that the system is operationally effective or suitable for combat, nor is it indicative of a level of system maturity that justifies full-rate production.

Joint STARS failure to meet its maintainability criteria during an operation less demanding than combat, even with such significant contractor involvement beyond that planned for in combat, also raises the question of the Air Force’s ability to develop a cost-effective maintenance plan for the system. This issue is recognized in the Under Secretary’s acquisition decision memorandum approving Joint STARS full-rate production. In that memorandum, the Under Secretary called for the Air Force to fully examine Joint STARS affordability, sustainability, and life-cycle costs, including the scope of contractor support.

System Sustainability and Suitability Is Questionable

“If it is determined that the system will be operated at rates similar to AWACS [Airborne Warning And Control System], it is questionable whether the [Joint STARS aircraft] can be sustained over time. Airframe problems have already been experienced on the existing [Joint STARS airframes], including a hydraulics failure and a cracked strut in the fuselage between the wings.”

In discussing the Joint STARS aircraft engines, DOT&E noted that they “are 1950s technology and may not be reliable” and cited AFOTEC’s reporting that engine failures were among the principal reasons that the aircraft failed to meet the break rate criteria during Operation Joint Endeavor. “. . . would face operational challenges taking off from five runways in Korea, each approximately 9,000 feet long. Operations out of Korea would likely require taking off with less fuel and subsequent aerial refueling or shortening the time on station.”

Another area of Joint STARS suitability concern is the system’s growth potential. DOT&E has reported that it is not clear that the remanufactured 707 platforms will be capable of incorporating all of the planned upgrades, noting that the airframe limits the system’s growth potential both in weight and volume. It reported that as the current mission equipment already fills much of the fuselage, there is little room for expansion. DOT&E also noted that increasing the payload weight would require longer takeoff runways or taking off with less fuel, thus increasing the aerial refueling requirement or decreasing mission duration.

DOT&E also noted that the system’s current computers limited its growth potential due to their having very little reserve processor time or memory. It stated that the Air Force requires that no more than 50 percent of central processor unit cycles or memory be utilized by a new system. DOT&E reported that “None of the E-8C computer subsystems meet these requirements.” It provided an example of the problem, stating that “the memory reserve of the operator workstations still does not meet the requirement, even after being increased from 128 megabytes to 512 megabytes just prior to .” This assessment is another indicator of the program’s elevated risk. As DOT&E noted “Future software enhancements and modifications may require significant hardware upgrades. . . .”

Operational Software Rated Significantly Immature

The AFOTEC report specifically pointed to the lack of maturity in Joint STARS software. For example, AFOTEC reported that “during Joint STARS , software deficiencies were noted on every E-8C subsystem;” the software “does not adequately support operator in executing the mission;” and “Joint STARS software does not show the expected maturity trends of a system at the end of development.”

In discussing Joint STARS software maturity, DOD advised us that the AFOTEC report judged the system overall operationally effective and suitable. Specifically, in reference to software problems, DOD stated that “the majority of software faults that occurred during Operation Joint Endeavor were resolved while airborne in less than 10 minutes.” However, both AFOTEC and DOT&E had some critical concerns regarding how Joint STARS software functioned. For example, according to AFOTEC, the “Joint STARS software is immature and significantly impedes the system’s reliability and effectiveness,” and according to DOT&E “Immature software was clearly a problem during [Operation Joint Endeavor]. . .” “. . .the prime contractor had to be called in to assist and correct 69 software-specific problems during the 41 E-8C missions . . . .an average of 1.4 critical failures per flight. . .” “Communications control was lost on 69 percent of the flights.” “The system management and control processor failed and had to be manually reset on half of the flights.”

DOD has stated that the Air Force “plans several actions to mature the software and provide the required support resources” and that “an interim software release in April 1997 will correct some software deficiencies identified during the operational evaluation.” DOD also noted that software updates will be loaded each year thereafter and that software changes are easily incorporated. How easily these software changes are incorporated remains to be seen because much of this software, according to AFOTEC and DOT&E, is poorly documented. For example, AFOTEC has reported that there are 395 deficiency reports open against the Joint STARS program, 318 of which are software related. DOT&E also stated that the more than 750,000 lines of Joint STARS software code are “poorly documented” and later commented that “Software problems with the communications and navigation systems were never fully corrected, even after extensive efforts by the system contractor.” These facts in combination with DOD’s comments raise the serious question as to which software deficiencies are to be addressed in the planned April software update.

Alternative Aircraft Should Be Considered

There is an opportunity not currently under consideration that could reduce the Joint STARS program cost and result in an improved system. Since the Joint STARS was approved for LRIP, the procurement cost objective of the Air Force’s share of the Joint STARS has increased by about $1 billion. Program costs escalated from approximately $5.2 billion to approximately $6.2 billion in then-year dollars. A DOD official informed us that of the $1 billion cost growth, $760 million is attributed to the increased cost to buy, refurbish, and modify the used 707 airframes to receive the Joint STARS electronics. The remaining cost growth is attributed to other support requirements and growth in required spare parts.

At least as early as 1992, the Boeing Company proposed putting Joint STARS on newer Boeing 767-200 Extended Range aircraft, but this proposal was not accepted as cost-effective. According to the 1996 Boeing price list, the commercial version of this aircraft can be bought for between $82 million and $93 million depending on options chosen (this is flyaway cost—the cost of a plane ready to be flown in its intended use). Furthermore, the flyaway cost of a commercial Boeing 757, which a Boeing representative informed us is in many respects more comparable to the 707s being used, is listed at between $61 million to $68 million. The actual cost of procuring either of these aircraft could be lowered by volume discounts and by the cost of the commercial amenities not required. On the other hand, these aircraft would require modifications to receive Joint STARS equipment, which would raise their cost.

DOD informed us that the cost of procuring, refurbishing, and modifying the current 707 aircraft to receive Joint STARS equipment is now estimated to be about $110 million per airframe. The cost of procuring and preparing new aircraft might be comparable or even less than the current cost. In addition, the Air Force would acquire a new platform that could have (1) greater room for growth (both volume and weight), (2) take off capability from a shorter runway, (3) greater time-on-station capability, (4) significantly improved fuel efficiency, (5) extended aircraft life over the 707 currently used, and (6) reduced operational and support cost.

In commenting on a draft of this report, DOD stated that it considered alternatives to the current air platform, both before LRIP started and at the full-rate production decision point. It also stated that the cost of moving the Joint STARS mission to an alternative platform would outweigh the benefits. We note, however, that at a meeting with DOD and service officials to discuss that draft, we asked about the reported DOD and service analyses. One Air Force official stated that the Air Force’s platform choice was not revisited prior to the full-rate production decision. None of the other 13 DOD and service officials present objected to that statement. Furthermore, when we asked for copies of the air platform analyses that were done in support of either the low-rate or the full-rate production decision, DOD was unable to supply those analyses. Finally, DOD officials have informed us that a Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance Mission Assessment has been performed that indicates that the Air Force could acquire a more effective system while saving $3 billion through the year 2010 by moving the Joint STARS mission to either a business jet or an unmanned aerial vehicle following the procurement of the twelfth current version Joint STARS aircraft.

Production Commitment Unnecessary and Risky

We have previously informed DOD of our concerns about the decision to move to full-rate production in spite of the numerous testing deficiencies reported by both AFOTEC and DOT&E. DOD responded that in making the decision to move to full-rate production, it “considered the test reports (both the services’ and the Director, Operational Test and Evaluation’s), the plans to address the deficiencies identified during developmental and operational testing, cost estimates, operational requirements, and other program information.”

Although DOD believes that “none of the deficiencies identified are of a scope that warrants interrupting production,” the production decision memorandum clearly reflects a recognition that this program carries significant risk. In his memorandum, the Under Secretary of Defense for Acquisition and Technology directed (1) an update of the Joint STARS Test and Evaluation Master Plan to “address multi-service [operational test and evaluation] deficiencies (regression testing);” (2) acceleration of the objective and threshold dates for the planned Follow-on Operational Test and Evaluation; and (3) the Air Force to “fully examine [Joint STARS] affordability, sustainability, and life cycle costs including the scope of contractor use for field-level system support.” “I am writing to be sure you know that the President is personally committed to [Joint STARS], has engaged Chancellor Kohl on this issue and will continue his personal involvement with key allies to ensure our goal is achieved. I would ask that you underscore your personal support for our collective efforts on behalf of [Joint STARS] when you meet with your NATO and European counterparts.”

Notwithstanding DOD’s September 1996 commitment to full-rate Joint STARS production, a DOD official informed us that the NATO armament directors in their November 1996 meeting delayed for 1 year any decision on designating Joint STARS as NATO’s common system or pursuing an alternate system to be developed.

Follow-on Operational Test and Evaluation Planned

In the process of moving the Joint STARS program forward into full-rate production, DOD produced a Beyond LRIP report for Congress and thus moved past a key congressional reporting requirement that serves as an important risk management mechanism. The Beyond LRIP report to Congress that is required before major defense acquisition programs can proceed into full-rate production serves to inform Congress of the adequacy of the operational testing done on the system and to provide it with a determination of whether the system has demonstrated effectiveness and suitability. Having issued this report, DOT&E is under no further obligation to report to Congress at the Beyond LRIP report level of detail on the adequacy of the operational testing or on whether the system has demonstrated effectiveness and suitability for combat. However, DOD plans follow-on test and evaluation of the system to address the deficiencies identified during the system’s earlier testing.

On September 20, 1996, DOT&E sent to Congress a Joint STARS “Beyond LRIP” report that (1) clearly indicates that further operational testing is needed, (2) could only declare effectiveness for operations other than war, and (3) stated that Joint STARS is unsuitable as tested. On September 25, 1996, DOD approved the full-rate production of Joint STARS. In the acquisition memorandum approving Joint STARS full-rate production, the Under Secretary of Defense for Acquisition and Technology called for an accelerated follow-on operational test and evaluation of Joint STARS that is to address the deficiencies identified in the initial operational test and evaluation DOT&E reported on in the Beyond LRIP report to Congress. The planned follow-on operational test and evaluation will provide an opportunity to judge the Joint STARS program’s progress in resolving the issues identified in earlier testing.

Conclusions

Notwithstanding any concurrent efforts to have Joint STARS designated as a NATO common system, Joint STARS test performance and the clearly unresolved questions about its operational suitability and affordability should have, in our opinion, caused DOD to delay the full-rate production decision until (1) the system had, through the planned follow-on operational test and evaluation, demonstrated operational effectiveness and suitability; (2) the Air Force had completed an updated analysis of alternatives for the Joint STARS to address the identified aircraft suitability and cost issues; and (3) the Air Force had developed an analysis to determine whether a cost-effective maintenance concept could be designed for the system. Furthermore, as they were judged “essential” to mission accomplishment and needed “to achieve a capable Joint STARS system,” the satellite communications and deployable ground support station features (present, but untested, during Operation Joint Endeavor) should also be tested during the planned Joint STARS follow-on operational test and evaluation.

Concerns of the magnitude discussed in this report are not indicative of a system ready for full-rate production. The program should have continued under LRIP until the issues identified by AFOTEC and DOT&E were resolved and the system was shown to be effective and suitable for combat. Furthermore, the recent cost growth related to refurbishing and modifying the old airframes being used for Joint STARS and questions regarding the suitability of those platforms indicate an opportunity to reduce the program’s cost and improve the systems acquired. We believe, therefore, that an updated study of the cost effectiveness of placing Joint STARS on new, more capable aircraft is warranted.

Recommendation

We recommend that the Secretary of Defense direct the Air Force to perform an analysis of possible alternatives to the current Joint STARS air platform, to include placing this system on a new airframe.

Matters for Congressional Consideration

Because of (1) DOD’s decision to commit to full-rate production in the face of the test results discussed in this report and (2) its subsequent decision to do additional tests while in production to address previous test deficiencies, we are convinced that DOD plans to proceed with the program. However, if Congress agrees that there is unnecessarily high risk in this program and believes the risk should be reduced, it may wish to require that:

The Air Force obtain DOT&E approval of a revised test and evaluation master plan (and all plans for the tests called for in that master plan) for follow-on operational testing to include adequate coverage of gaps left by prior testing and include testing of any added features considered part of the standard production configuration and that DOT&E considers key system components.

DOT&E provide a follow-on test and evaluation report to Congress evaluating the adequacy of all testing performed to judge operational effectiveness and suitability for combat and a definitive statement stating whether the system has demonstrated operational effectiveness and suitability.

DOD develop and provide Congress an analysis of alternatives report on the Joint STARS air platform that considers the suitability of the current platform and other cost-effective alternatives, and the life-cycle costs of the current platform and best alternatives.

Agency Comments and Our Evaluation

In commenting on a draft of this report, DOD disagreed with our recommendation that the Air Force be directed to perform an analysis of possible alternatives to the current Joint STARS air platform. It also disagreed with our suggestion that Congress may wish to require DOD to develop and provide Congress a report on that analysis. DOD stated that alternative platforms were considered prior to both the start of LRIP and the full-rate production decision. DOD stated that based on (1) the fact that over half the fleet is already in the remanufacturing process or delivered to the user; (2) the large nonrecurring costs that would be associated with moving the Joint STARS mission to a different platform; (3) the additional cost to operate and maintain a split fleet of Joint STARS airframes; and (4) the expected 4-year gap in deliveries, such a strategy would force the costs of moving the Joint STARS mission to a different platform outweigh the benefits.

DOD’s comment about having previously considered alternative platforms is inconsistent with the information we developed during our review and with Air Force comments provided at our exit conference. In an effort to reconcile this inconsistency, we requested copies of the prior analyses of alternative platforms, but DOD was not able to provide them. DOD’s statement that the costs of moving the Joint STARS mission to another platform would outweigh the benefits contradicts Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance Mission Assessment briefings given the Quadrennial Defense Review. Those briefings recommend (1) limiting Joint STARS production to 12 aircraft, (2) moving the Joint STARS mission to either corporate jets or Unmanned Aerial Vehicles, and (3) phasing out Joint STARS 707 variants as quickly as the new platform acquisitions will allow. According to those briefings, implementation of this recommendation could result in a more effective system and save over $3 billion through fiscal year 2010. We believe that the issue clearly warrants further consideration. Furthermore, given DOD’s resistance to the concept, we are more convinced of the merits of our suggestion that Congress might wish to require a report on such an analysis.

In commenting on our draft report, DOD also indicated that congressional direction was unneeded on our suggestions that Congress might wish to require (1) DOT&E approval of a revised test and evaluation master plan for the planned Joint STARS follow-on operational test and evaluation and (2) DOT&E to provide Congress with a follow-on operational test and evaluation report on the adequacy of Joint STARS testing and stating whether Joint STARS has demonstrated operational effectiveness and suitability. DOD stated that congressional direction on the first point was unneeded because the Joint STARS full-rate production decision memorandum required that the test and evaluation master plan be updated for Office of the Secretary of Defense approval and current DOD policy is that DOT&E will review, approve, and report on oversight systems in follow-on operational test and evaluation. DOD also stated that congressional direction on the second point is unneeded because DOT&E has retained Joint STARS on its list of programs for oversight and is to report on the system in its annual report to Congress as appropriate.

DOD’s response did not directly address our point since, as DOD pointed out, the acquisition decision memorandum that approved full-rate production required Office of the Secretary of Defense approval, not DOT&E approval, of the follow-on operational test and evaluation master plan. During the course of our review, DOD officials informed us that there was significant disagreement between the Air Force and DOT&E as to what follow-on testing was needed. It was indicated that the issue would probably have to be resolved at higher levels within the department, an indication of greater flexibility than DOD implies. Furthermore, while DOD stated there were some improvements and enhancements “that could benefit the warfighter” and acknowledged that those features were not tested, it did not respond to our comments that DOT&E judged those features “essential” to mission accomplishment or commit to their operational test and evaluation. Given these facts, we have not only maintained our suggestion that Congress may wish to require the Air Force to obtain DOT&E approval of a revised test and evaluation master plan, but also strengthened it to include DOT&E approval of supporting test plans.

In its response to our suggestion that Congress may wish to require that DOT&E provide it a detailed, follow-on test and evaluation report, DOD states congressional direction is unnecessary as DOT&E will report on the system, among many others, in its annual report to Congress. DOD’s comment fails to recognize, however, that we are suggesting that, given the already reported test results, Congress may wish a more detailed report outlining the adequacy of and the system’s performance during follow-on operational testing to help in its oversight and provide it assurance that the system’s problems have been substantially resolved. Given that (1) Congress felt such reporting to be beneficial enough to require it before a system can proceed beyond LRIP and (2) the fact that DOT&E, in the required report provided for Joint STARS, could not certify effectiveness for war and found the system unsuitable as tested, we continue to believe that Congress may wish to require a similar report based on the follow-on operational test and evaluation planned.

DOD’s comments are reprinted in their entirety in appendix I, along with our evaluation.

Scope and Methodology

To determine whether Joint STARS test performance indicates a maturity justifying full-rate production, we interviewed officials and reviewed documents in Washington, D.C., from the DOD Office of the Director of Operational Test and Evaluation and the Joint STARS Integrated Product Team. We reviewed the Air Force Operational Test and Evaluation Center’s multi-service operational test and evaluation plan and its final report on that testing and the DOD Director of Operational Test and Evaluation’s Beyond LRIP report. To determine whether DOD considered and resolved important cost and performance issues prior to making its full-rate production decision, we reviewed Joint STARS program budget documents and program-related memoranda issued by the Under Secretary of Defense for Acquisition and Technology. To determine whether it is possible that a more useful operational test and evaluation report can be provided Congress, we reviewed the statute governing operational testing and evaluation, examined DOT&E’s Beyond LRIP report, and considered other relevant program information. We considered and incorporated where appropriate DOD’s response to our September 20, 1996, letter of inquiry and its response to a draft of this report. We conducted this review from October 1996 through April 1997 in accordance with generally accepted government auditing standards.

We are sending copies of this letter to other appropriate congressional committees; the Director, Office of Management and Budget; and the Secretaries of Defense, the Army, and the Air Force. Copies will also be made available to others upon request.

If you or your staff have any questions, please contact me, Mr. Charles F. Rey, Assistant Director, or Mr. Bruce Thomas, Evaluator-in-Charge, at (202) 512-4841.

Comments From the Department of Defense

The following are GAO’s comments on the Department of Defense’s (DOD) letter dated March 31, 1997.

GAO Comments

1. We have not suggested or recommended that Joint STARS production be interrupted. We have, however, suggested actions that we believe (1) will help reduce the program’s risk; (2) could result in the acquisition of a more effective, less costly system; and (3) could help decisionmakers ensure that the Joint STARS program continues to make progress. 2. The report has been modified in light of DOD’s comments. 3. DOD’s indication that other factors were considered in deciding to proceed to full-rate production is a signal that DOD and the Air Force are willing to accept a high level of risk even when the Director, Operation, Test, and Evaluation (DOT&E) has concluded that the system was unsuitable as tested and operational effectiveness for war remains to be demonstrated. We believe, given the system’s test performance as reported by both the Air Force Operational Test and Evaluation Command (AFOTEC) and DOT&E and the program’s procurement cost growth of $1 billion between the low-rate and full-rate production decision points, that an informed full-rate production decision required the following information: (1) an approved test and evaluation master plan for follow-on operational testing and specific plans for the tests called for in that master plan, (2) the results of the already ongoing study of ways to reduce the program’s cost, and (3) an analysis of alternatives to the current platform. DOD did not have these items in hand when it made its decision. We must also note that DOD implies that our recommendations would require a break in production. This is inaccurate. As we stated in the body of our report, the program could have continued under low-rate initial production (LRIP) until operational effectiveness and suitability for combat were demonstrated and plans to address identified deficiencies and reduce program costs were completed. 4. In its report on the Joint STARS multi-service operational test and evaluation, AFOTEC stated that “Joint STARS software is immature and significantly impedes the system’s reliability and effectiveness.” We do not believe that, given the software intensive nature of the system, this statement supports a conclusion that the system could be judged operationally effective. 5. We must note that follow-on operational test and evaluation of the system was planned before the full-rate production decision. The full-rate production decision called for acceleration of that testing and for that testing to address deficiencies identified in the earlier tests. Joint STARS could have continued under LRIP pending a demonstration of operational effectiveness and suitability. 6. This speaks to the number of aircraft missions planned and the number for which an aircraft was provided. It does not address the quality or quantity of the support provided during those missions. Furthermore, DOD’s comment refers to the same—operation that is reported on in both the Air Force and DOT&E reports and in this report. 7. U.S.-based contractor support was utilized during the first Operation Joint Endeavor deployment. It is also our understanding that during the second Operation Joint Endeavor deployment the Air Force may have utilized a “reach-back” maintenance concept in which U.S. stationed contractor staff were providing field support through satellite communications. Moreover, DOD and Air Force officials told us that at least at the beginning of the second Operation Joint Endeavor deployment, contractor staff were flying on the deployed aircraft. This clearly raises the question of what the overall level of contractor support was for both the first and second deployments. “As already discussed, extensive efforts by the system contractor were required to achieve the demonstrated availability for the E-8C aircraft. Even with those efforts the system was not able to meet the user criteria for several measures directly related to the maintenance concept in place during —a concept that involved considerably more contractor support than previously envisioned.”

11. As we noted in the body of our report, Joint STARS failed to meet test criteria during an operation less demanding than combat, even with such significant contractor involvement beyond that planned for in combat. In discussing operational tempo in its Beyond LRIP report, DOT&E stated that if the system is operated at rates similar to the Airborne Warning and Control System, “it is questionable whether the [Joint STARS aircraft] can be sustained over time.” DOD commented that an unbiased assessment of the measure of Joint STARS’ ability to maintain the required tempo could not be made and would be tested during the follow-on operational test and evaluation. We believe that an informed full-rate production decision requires knowledge of a system’s ability to satisfy the operational tempo expected of it. DOD made its Joint STARS full-rate production decision without this knowledge. 12. We understand that Joint STARS, like most systems, has limitations that need to be planned around. At issue here is a question of how great those limitations are and whether they are acceptable. DOD states that “the user is satisfied that the system meets requirements.” However, we must note that the Air Force’s own Operational Test and Evaluation Center reported that the “two critical suitability [measures of performance, sortie generation rate and mission reliability rate], were affected by [Operation Joint Endeavor] contingency requirements and system stability problems.” “The high failure rate of aging aircraft components affected as critical failures were statistically determined to affect over 30 percent of the sorties flown. Analysis revealed the elevated critical failure rate was steady and showed no potential for improvement. Technical data and software immaturity affected the maintainability of the aircraft, and contractor involvement further compromised clear insight into the Air Force technicians’ ability to repair the system.”

AFOTEC also reported on Joint STARS performance relative to 15 supporting suitability criteria. It stated “Eight did not meet users’ criteria. One was not tested. Only one . . . met the users’ criteria. The remaining five are reported using narrative results.” 13. DOD discusses only the weight growth of funded activities, leaving open the question of whether there are future, but currently unfunded, improvements planned that will add weight growth. Air Force officials told us that the Airborne Warning and Control System had experienced weight growth over the life of its program. That growth was attributed to the system’s being given added tasks over time. We believe it reasonable to expect that the Joint STARS program experience might track that of the Airborne Warning and Control Systems program, that is, be given added tasks and face weight growth as a result. Also, regarding Joint STARS room for growth, DOD previously advised us that Joint STARS currently has about 455,000 cubic inches of space available. We must note that this equates to a volume of under 7 feet cubed and that in commenting on the system’s space limitation, DOT&E stated “There is little room available for additional people or operator workstations.” 14. As we stated in the body of our report, how easily these software changes are incorporated remains to be seen. 15. We requested and DOD provided additional information on this point. DOD’s subsequent response indicates that this DOD comment was in error. In its subsequent response, DOD stated that the follow-on test and evaluation was accelerated “to reflect desire for earlier to evaluate fixes to deficiencies.” We believe this statement reflects a recognition of increased program risk. 16. The acquisition decision memorandum approving Joint STARS production clearly indicates that the Skantze study mentioned was not completed at that time. We believe that the full-rate production decision should have been made with the Skantze study in hand. Furthermore, we do not understand why DOD felt the need to direct the Air Force to fund and implement a plan that is to save it money, but felt no need to direct the Air Force to examine alternative platforms that at least one other DOD panel had stated would not only save $3 billion but also provide greater effectiveness. 17. We believe that not only should DOT&E approval of the Joint STARS Test and Evaluation Master Plan be required, but also of all supporting test plans. We have changed the language of this matter for congressional consideration accordingly. 18. We are suggesting that Congress may wish to request a more detailed report, one at the Beyond LRIP report level of detail, a level of detail not provided in DOT&E’s annual report. Given that DOT&E could only state effectiveness for operations other than war—could not state a belief as to whether the system would be effective in two of the three critical operational roles it is expected to perform in war—and found the system unsuitable as tested, we believe that such report would help Congress maintain program oversight. DOD’s comment of “other reports as appropriate” leaves the matter in DOD’s hand to decide if Congress would benefit from such a report.

The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.