Defense Acquisitions: Changes in E-10A Acquisition Strategy Needed before Development Starts
Released: 2005-03-15T00:00:00

Letter

Background

The E-10A program comprises three primary elements: the aircraft, radar, and battle management command and control subsystem. The aircraft is a Boeing 767-400ER, the largest 767 variant Boeing makes. The Air Force has only contracted for one aircraft to date, because a final decision on the operational platform has not been made. This aircraft is a commercial product that will be modified for military use and used as a testbed. At this time there is only 1 unfilled order for the 400 model in the Boeing assembly line and 25 other unfilled orders for other smaller 767 models. If the Boeing production line were to close down before the Air Force is positioned to make a production decision on the E-10A it would have to find an alternative. Alternatives could include a different aircraft type or model or the purchase of 767-400ER aircraft from commercial airline companies.

The radar planned for the E-10A began development in 1997 as a response to the growing concern about cruise missile proliferation. Initially, it was intended to upgrade the radar on the Joint Surveillance Target Attack Radar System (Joint STARS). The upgraded radar was to have advanced sensor technology, providing air-to-air capability for cruise missile defense and significant increases in ground surveillance capability. Shortly after the program began development, the Air Force restructured the program to develop a modular, scalable radar suitable for use on a variety of airborne platforms. OSD approved the development of the multiple platform radar in 2003. It is being designed for inclusion on the Global Hawk and E-10A programs.

The Air Force began evaluating the need to improve its airborne battle management command and control capabilities in 2002. The planned E-10A battle management command and control subsystem is software intensive and intended to enable the E-10A to process and display sensor data from the radar and eventually from off board sensors so that the onboard crew can take actions against time sensitive targets. The Air Force issued a contract in September 2004 to begin preliminary design efforts for this subsystem.

We have a body of work focused on best practices in product development and weapon systems acquisition. This work has found that key to success is formulation of a business case that matches product requirements to available resources—proven technologies, sufficient engineering capabilities, time, and funding. Several basic factors are critical to establishing a sound business case for undertaking a new product development. First, the needs of the party seeking the new product, the user, must be accurately defined, alternative approaches to satisfying these needs properly analyzed, and quantities needed for the chosen system must be well understood. The developed product must be producible at a cost that matches the users’ expectations and budgetary resources. Finally, the developer must have the resources to design and deliver the product with the features that the customer wants when it is needed. If the financial, material and intellectual resources to develop the product properly are not available, development does not go forward. Additionally, an evolutionary and knowledge-based acquisition strategy that captures critical knowledge before key decision points in the program is needed to execute the business plan. This calls for a realistic assessment of risks and costs; doing otherwise undermines the intent of the business case and invites failure. Ultimately, preserving the business case and attaining critical knowledge in time for decisions strengthens the ability of managers to say “no” to pressures to accept high risks or unknowns.

If best practices are not followed, we have found a cascade of negative effects becomes magnified in the product development and production phases of an acquisition program. These have led to acquisition outcomes that included significant cost increases and schedule delays, poor product quality and reliability, and delays in getting the new capability to the warfigher. These outcomes have been demonstrated in other programs such as the F/A-22 fighter, C-17 airlifter, V-22 tiltrotor aircraft, PAC-3 missile, and others.

Uncertainties Exist in E-10A Business Case as It Approaches Its Decision Point

Questions remain as the Air Force develops the E-10A program’s business case to support the decision to begin development in April 2005. The DOD has identified a need for a cruise missile defense capability and the Air Force has selected the E-10A to meet this need. There are, however, unanswered questions in both the requirement and resource elements of the E-10A business case. OSD is still studying whether the E-10A is the most cost-effective alternative for the cruise missile requirement and the extent of battle management command and control needed on board to satisfy the intended need. Finally, assessments of the technology maturity, estimated costs, and funding availability are still in process.

OSD Is Still Studying the Need for the E-10A

OSD officials from the Program Analysis and Evaluation Directorate are not satisfied that the studies done by the Air Force to select the E-10A sufficiently analyzed alternative systems. As a result, they are reviewing alternative systems and attempting to determine the most cost-effective solution to satisfy the warfighter’s needs. OSD officials agree that the E- 10A could provide an increased capability in identifying and tracking ground moving and time-sensitive targets. However, they believe that if there are less costly systems that can provide similar capabilities, it could be more cost-effective to buy those systems. The Air Force began efforts in 1997 to develop a radar sensor that would detect cruise missiles as part of the Joint STARS program. The Air Force examined different size and power combinations for the radar and which platforms had the capacity to carry the radar and still perform multiple missions. These analyses assumed that only manned airborne platforms could meet these requirements. The Air Force completed a formal analysis of alternatives in February 2002 of different possible host platforms for the radar. The study indicated that other aircraft could meet many of the requirements but were based on older commercial technology that was less efficient to operate. The Air Force analysis concluded that the Boeing 767-400ER was the optimal choice given the future multi-mission purpose of the system, and the size, weight, and performance requirements of the radar.

OSD officials are also uncertain about the degree of battle management command and control capability needed onboard the E-10A versus transmitting the information gathered by the E-10A to other command and control centers. According to the Air Force, the need for an onboard capability is driven by the large amounts of data that would be collected and analyzed, the limited bandwidth to transmit the data, and the need to have line-of-sight communications for time-sensitive targeting, particularly against cruise missiles. OSD officials said they are looking at whether the battle management subsystem has to be part of the E-10A platform to meet the timelines identified by the Air Force. They expect to present their results by March 2005. Air Force officials told us that some of the battle management functions are currently performed by ground units, but these ground units cannot adequately respond to real-time events involving moving targets like cruise missiles. The E-10A’s primary function will be battle management command and control of cruise missile detection and time-sensitive targeting activities. As a result, its battle management capabilities will be tailored to support those functions. These capabilities were validated in October 2004 by the Joint Requirements Oversight Council in preparation for the program’s upcoming Milestone B decision.

To provide these capabilities, an onboard crew will be required. The current E-10A crew size is estimated at 27 staff—2 flight crew, 21 mission operators, and 4 technicians. According to the Air Force, the crew size could change depending on the mission and the degree of automation on the system. However, the Air Force has not performed any incremental analysis to show crew size for individual specific missions, such as doing cruise missile defense only.

Assessments of Needed Resources to Develop and Produce the E-10A Are Incomplete

To date, the Air Force has not identified sufficient or available resources to meet the warfighter’s requirements and to start the development program. The Air Force program office has completed its assessments of E-10A critical technologies, cost estimates, and funding needs but these assessments are being reviewed by OSD. While some resources will meet the requirements, others are either unproven or in a state of flux. Radar development started under a separate program, the Radar Technology Insertion Program, and most radar technologies were reported as mature. Because the Air Force did not provide GAO its technical assessment of the battle management command and control system critical technologies, we consider the maturity levels unproven, even though program officials told us these technologies meet minimum maturity standards. In addition to technologies, the financial resources for the program are in a state of flux. The E-10A cost estimate for development and production is still a work in process and funding was recently reduced by $600 million for fiscal years 2006 and 2007, which according to DOD officials will substantially impact the program.

Status of Technologies

Most radar technologies are at a high level of maturity, but evidence was not provided to support stated maturity levels of the battle management command and control subsystem. The Air Force assessed radar technologies prior to the October 2003 start of the Radar Technology Insertion Program. The critical technologies identified in the radar improvement program included the radar architecture, modes, receiver/exciter, and signal processor among others. Of the nine technologies identified, six were assessed as mature to our best practice standard; the remaining three were one level below the best practice requirement for mature technologies, a level DOD policy states is sufficient to begin development. These three technologies are the pulse compression unit, the structure, and the modes. Since the 2003 radar technology assessment, the radar improvement program completed its final design review in June 2004. Numerous tests have been conducted on small-scale radar prototypes to mitigate program risks. These tests electronically drove a signal through the radar, demonstrating the basic functionality of the design. However, the radar subsystem being designed for the E-10A has demonstrated neither form nor fit, nor has it been integrated on the aircraft platform. Although the integration process is an inherently high-risk endeavor, Air Force officials stated they have a process in place to manage these risks. The actual size of the E-10A’s radar will be significantly larger than the tested prototype and will require the E-10A testbed aircraft in order to complete the demonstration currently scheduled to occur in 2010. The process of scaling the radar to the appropriate size and ensuring that all the individual modules work together has yet to be accomplished. Recognizing this, program officials have identified the integration of the radar as a critical technology for the E-10A weapon system. The level of this technology’s maturity has not yet been finalized. OSD officials accepted the Air Force’s assessment of the radar technologies but expect more detailed information on the technologies when the E-10A weapon system undergoes its Milestone B review in April 2005.

An assessment of the battle management command and control subsystem technologies was not provided for our review. This subsystem is complex and software intensive. E-10A program officials told us these technologies would meet the minimum DOD standard for starting a program. However, the Air Force only recently directed the contractor to begin systems engineering efforts to determine a preliminary design for this subsystem. Development of critical software needed to demonstrate the technologies has not started. The first increment of software is not scheduled to be delivered until January 2008. On other major weapon system development programs, we have found software development to be a substantial cause for delays in technology development, system deliveries, and increased costs. Therefore, even though program officials have stated technologies are sufficiently mature, we think stronger evidence will be needed to demonstrate their claim.

Status of Cost Estimate and Funding Needs

The Air Force has completed its cost estimate for the total E-10A program and released it to OSD for review. The cost estimates for each of the three major program elements contain risk. The biggest area of cost uncertainty is the battle management command and control subsystem. It is a highly complex software-intensive system. A contract was issued in September 2004 for about $71 million to begin early design and engineering efforts to support a preliminary design review in late 2005. Until this initial design and engineering effort is completed, the program will not be able to establish high confidence in its estimated costs. In addition, the aircraft contract only calls for the delivery of one commercial 767-400ER for testing. To convert this aircraft to military use, there will be additional costs for installing communication antennas, a refueling receptacle, hull hardening, and FAA airworthiness certification. According to the Air Force, these costs have been factored into its latest program estimate. The initial cost estimate for the radar program, managed separately from the E-10A program, has grown. Prior to entering system development, OSD determined that projected costs were understated and directed the Air Force to increase its funding by $154 million.

The Air Force acknowledges that funding for the E-10A program is also a major concern. Funding cuts have delayed its start. It has undergone two congressional budget reductions; the first cut in fiscal year 2003 ($343 million) required a significant program replanning effort. The second cut in fiscal year 2005 ($115 million) resulted in schedule delays for the planned test program, system integration lab, testbed aircraft delivery, and the E-10A’s first flight. The Air Force states these cuts have caused the planned initial operating capability date to slip 3 years to 2015. A third cut, recently proposed by OSD in December 2004, reduces the program’s budget request by $300 million in both fiscal year 2006 and 2007—a total reduction of $600 million. The program office is in the process of evaluating the impact of these reductions and officials indicated that because these represent a reduction of about 45 percent in each year, they will have a significant impact on the program if they are sustained. OSD officials indicated that efforts related to aircraft development and the delivery of the test aircraft will likely bear the bulk of the reductions. This will have an impact on planned program milestones. They said it was important to keep the radar program funded because it is developing the radar planned for the new Global Hawk unmanned aerial vehicle in addition to the E-10A.

E-10A Acquisition Strategy Does Not Capture Critical Knowledge before Program Decision Points

The E-10A acquisition strategy raises concern as key decisions are planned before critical product knowledge is available. For example, the strategy for developing the first E-10A increment does not allow for adequate integration or prototype demonstration to ensure the design is stable at the system critical design review. System integration allows program officials to measure the stability of a product’s design and its ability to meet established requirements. Both commercial companies and DOD recognize the attainment of this knowledge as being demonstrated by the completion of most engineering drawings and some demonstration of the system level capabilities in a prototype. A stable design that meets requirements should be achieved by critical design review, before system demonstration and initial manufacturing of production representative products begins. However, the Air Force does not expect to deliver the battle management command and control and radar subsystems to the integration laboratory until 2008 and 2009, after critical design review, scheduled in 2007. The transition of the battle management command and control and radar subsystems from the integration lab to the 767-400ER test airframe is not scheduled to begin until late-2009, nearly 2 years after the critical design review and only a few months prior to the program’s production commitment decision. As a result, critical knowledge about the basic performance of key subsystems integrated into an actual E-10A prototype will not occur until 2010 (see fig. 1).

Additionally, the fully integrated E-10A prototype will not be available for testing prior to the scheduled decision to begin production. This strategy requires significant concurrency among the technology development, product development, and production phases and places decision makers at a disadvantage by not knowing if the E-10A can demonstrate it meets system performance and reliability requirements before transitioning into production. In fact, the results of operational testing are not scheduled to be available until four of the six planned E-10As are already in production in 2011, greatly increasing the risks of costly design changes and schedule delays later in the program (see fig. 2). Our past reviews have found this to be a high-risk acquisition approach.

The Air Force is planning to use an incremental approach to achieve the E-10A’s full capability with each subsequent increment adding capability. Although an incremental approach can reduce risks, the failure to capture critical knowledge while developing the first increment will likely reduce the benefits of such an approach. As currently planned there will be four distinct E-10A increments. Program officials are planning to conduct major program decision reviews prior to beginning development and demonstration of each increment. This approach, if implemented as planned, will provide decision makers with an opportunity to review the program’s progress and risk before making further investment decisions thus reducing risk in the program. The first increment is expected to provide the users with many of the system’s basic required capabilities. Those capabilities include cruise missile defense and on-board command and control capability for processing, displaying, and communicating the data needed to address time-sensitive targets. Subsequent increments will enhance the system’s capabilities, moving them closer to objective levels by increasing the amount of data processing and analysis done by computers and decreasing the amount done by human analysts with computer assistance, thus shortening the time it takes to make decisions. However, if the first increment falters, the Air Force will likely spend increasing amounts of time and money to achieve this initial capability, thereby delaying subsequent increments.

Conclusions

The current conditions surrounding the development of the E-10A business case portend the potential for poor outcomes if requirement, resource, and acquisition strategy deficiencies are not resolved before system development and demonstration begins. The decision to start a major weapon systems acquisition program for the E-10A requires an executable business case that demonstrates the E-10A is the best way to satisfy the gap in warfighter’s capability and that the concept can be developed and produced within existing resources. An evolutionary and knowledge-based acquisition strategy is needed to ensure this business case can be executed within planned goals. The Air Force and OSD are still determining if a sound business case exists. Questions still surrounding the business case include: Is the E-10A the most cost-effective alternative?

How extensive of a battle management command and control capability is needed?

Are technologies at a high level of maturity?

Is there sufficient funding to develop and deliver the capability in time?

The acquisition strategy also fails to capture critical design, manufacturing, and reliability data in time to make investment decisions for moving the program through the development program into production. The gaps in knowledge increase the likelihood that the Air Force will not be able to deliver on the cost, schedule, and performance goals in its business case.

Recommendations for Executive Action

Because gaps exist in the information needed to make a sound business case to start a major acquisition program, we recommend that the Secretary of Defense ensure that the open business case questions are answered before a decision is made to start the E-10A program. Additionally, to ensure a greater likelihood of success, if the E-10A program is approved to begin, we recommend the Secretary direct the Air Force to revise the acquisition strategy to ensure sufficient time is included in the schedule to (1) integrate and demonstrate the design before moving past the critical design review and (2) test a production representative E-10A prototype before starting production.

Agency Comments and Our Evaluation

DOD provided us with written comments on a draft of this report. The comments appear in appendix II.

DOD concurred with our recommendation that the Secretary ensure that the open business case questions are answered before a decision is made to start the E-10A program. DOD provided some information on the current status of these questions and implies that some of the business case questions had been answered. We believe that until the OSD/Program Analysis and Evaluation study is completed and final results are provided to OSD acquisition decision makers, the business case questions remain open.

DOD partially concurred with our recommendation that the Secretary direct the Secretary of the Air Force to revise the E-10A acquisition strategy to ensure sufficient time is available to (1) integrate and demonstrate the design before moving past the critical design review and (2) test a production representative E-10A before starting production.

Regarding (1), DOD stated that OSD policy does not require the integration and demonstration of a design before critical design review. We disagree. Section E1.1.14 of Department of Defense Directive 5000.1, The Defense Acquisition System, states that “PMs…shall reduce integration risk and demonstrate product design prior to the design readiness review.” DOD’s design readiness review is required to end the system integration phase of system development and demonstration. Additionally, DOD’s entrance criterion for the demonstration phase requires a demonstration of the integrated product in a prototype. Nonetheless, DOD stated that it is restructuring the program with the goal of demonstrating the radar and battle management technologies in a prototype before starting systems development and demonstration. This approach incorporates the knowledge-based approach inherent in commercial best practices and endorsed by DOD policy. In its comments, DOD acknowledges that this approach will increase confidence in the program’s cost estimate and allow time to evaluate the aircraft platform.

Regarding (2), DOD stated that the Milestone C production decision for low rate initial production decision will be based on the initial test results from a representation E-10A aircraft system. While the program schedule in effect at the time of our review did not indicate this, we believe this approach is more consistent with a knowledge-based acquisition strategy. By testing a production representative aircraft prior to committing to production, DOD will be able to reduce program risks and make informed decisions based on actual system capabilities and performance information.

DOD also provided technical comments to our report. We made changes where appropriate but many of these comments were based on a new acquisition strategy that plans to delay the E-10A program Milestone B decision until 2010. We did not make DOD’s recommended changes to the report that reflected this new schedule because it has not been approved and we have not had the opportunity to review it.

We are sending copies of this report to the Secretary of Defense and the Secretaries of the Air Force, the Army, and the Navy. We will also provide copies to others on request. In addition, the report will be available at no charge on the GAO Website at http://www.gao.gov.

Please contact me at (202) 512-4841 if you have any questions concerning this report. Other key contributors to this report were Martin Campbell, Michael Hazard, Travis Masters, Rae Ann Sapp, David Schilling and John Krump.

Appendix I: Scope and Methodology

During our review we discussed the E-10A program with officials from the following organizations in the Office of the Secretary of Defense, Undersecretary of Defense for Acquisition Technology and Logistics; the Director, Defense Systems/Developmental Test and Evaluation; the Director, Operational Test and Evaluation; the Director, Defense Research and Engineering; and the Director, Program Analysis and Evaluation. We also discussed the E-10A with the technical director of the Joint Theater Air Missile Defense Organization. In addition, we discussed the program with officials from several organizations in the Air Force. These officials included representatives from the Information Dominance Directorate with the Office of the Assistant Secretary for Acquisition; the Directorate of Operational Requirements; the Command Control Communications Intelligence and Reconnaissance Center at Langley Air Force Base; the Electronic Systems Center at Hanscom Air Force Base; and the Aeronautical Systems Center at Wright Patterson Air Force Base.

To determine the progress the Air Force had made in developing the business case for the E-10A, we obtained available information on the system’s requirements and resources. However, the information we received on resources such as technology maturity, cost, funding, quantities, and schedule was limited. We discussed this information with knowledgeable program office and oversight officials. We also contacted officials studying force structure issues that could impact the requirements for the E-10A program. To assess the validity of the proposed business case, we compared the E-10A information with best commercial practices and DOD policy guidance for new development programs. Because the E-10A program has not yet been approved to enter system development and demonstration, specific information on the system’s technology readiness assessment and total program cost and funding were not available. As a result, we could not conduct a detailed assessment of these elements of the business case. However, because of other related information, such as the status of the software intensive battle management command and control subsystem, the significant reduction in funding for fiscal years 2006 and 2007, and the ongoing studies to answer OSD concerns, we were able to conclude that at the time of our review key business case elements were still not mature enough to begin product development. For example, complex and software intensive subsystems in other programs have caused major problems that have delayed achieving technology maturity and the Air Force has only recently directed the contractor to begin early systems engineering effort to determine a preliminary design for the E-10A battle management subsystem. Additionally, the $600 million reduction in funding planned for the first 2 years will almost certainly require the program to extend its planned schedule resulting in additional costs and funding requirements not yet estimated. These are business case elements that need to be firmly established before entering the upcoming Milestone B decision point.

To determine the soundness of the E-10A’s acquisition strategy, we obtained available information on the program’s original and revised acquisition plans from the program office and discussed it with functional oversight and program officials. In addition, we compared the E-10A’s planned strategy to best commercial practices and DOD’s knowledge- based acquisition policy. However, since our analysis, the program’s budget request was reduced by a total of $600 million in fiscal years 2006 and 2007.

We conducted our review from January 2004 to January 2005 in accordance with generally accepted government auditing standards.

Appendix II: Comments from the Department of Defense