Comanche Helicopter: Testing Needs to Be Completed Prior to Production Decisions
Released: 1995-05-18T00:00:00

Letter

Background

The Comanche program was established in 1983 to replace the Army’s light helicopter fleet. The contractor team of Sikorsky Aircraft Corporation and Boeing Helicopter Company were expected to design a low-cost, lightweight, advanced technology helicopter capable of performing the primary missions of armed reconnaissance and attack. Critical to achieving these capabilities are the successful development of advanced technologies, including composite materials, advanced avionics and propulsion systems, and sophisticated software and hardware. The Army must meet ambitious maintainability goals in order to (1) realize significantly lower operating and support costs predicted for this program and (2) achieve a wartime operational availability for the Comanche of 6 hours per day.

In December 1994, the Secretary of Defense directed the Army to restructure the Comanche helicopter program as part of efforts to meet budgetary constraints. The Secretary’s restructure decision reduced funding for the program from $4.2 billion to $2.2 billion for fiscal years 1996 through 2001. In addition to extending the development phase by 3 years, it also called for two flyable prototypes to be produced and the Comanche production decision to be deferred.

In response to the Secretary’s decision, the Army proposed a program restructure that would allow it to acquire, within the Secretary’s funding constraint, six aircraft in addition to the two prototypes by deferring developmental efforts to fiscal year 2002 and beyond. DOD approved the proposal in March 1995.

Restructuring Continues Risks

The Army’s restructuring of the Comanche program continues risks (1) associated with making production decisions before knowing whether the aircraft will be able to perform as required and (2) of higher program costs.

Restructuring Continues Highly Concurrent Program

According to DOD’s April 1990 guidelines for determining degrees of concurrency, a program with high concurrency typically proceeds into low-rate initial production before significant initial operational test and evaluation is completed. Regarding the need to keep concurrency low, the guidelines note that establishing programs with no concurrency, or a low degree of concurrency, avoids the risks that (1) production items have to be retrofitted to make them work properly and (2) system design will not be thoroughly tested. As we recently reported, aircraft systems, including the T-45A and C-17, that entered low-rate initial production before successfully completing initial operational testing and evaluation experienced significant and sometimes costly modifications to achieve satisfactory performance. Under the Army’s restructured program, operational testing will not begin until after the low-rate initial production decision is made, continuing the risks associated with the highly concurrent Comanche program.

In responding to the Secretary’s restructure decision, the Army proposed, and was subsequently granted approval, to buy six “early operational capability” aircraft, in addition to the two prototypes that were to be acquired under the Secretary’s decision. According to program officials, these aircraft are estimated to cost in excess of $300 million. The Army does not consider these aircraft as either prototype or low-rate initial production aircraft; however, program officials believe that when these aircraft are fielded, the Army will be able to better evaluate the Comanche’s mission capability. The Army intends to fund these aircraft by deferring additional developmental efforts to fiscal years 2002 and beyond.

Under the Army’s restructured program, operational testing will not begin until well after funds are committed to buy production aircraft. Armed reconnaissance and attack mission equipment packages are to be integrated into the six early operational aircraft by fiscal year 2004. The Army plans to use these aircraft to start operational testing by about August 2005. However, long-lead production decisions are scheduled for November 2003, and low-rate initial production is planned to start in November 2004, about 9 months before operational testing begins.

According to DOD’s guidelines, the amount of risk associated with concurrency can be limited by reducing production aircraft to the minimum necessary to perform initial operational testing. The Army maintains that under the stretched out program it can conduct initial operational testing with the six early operational aircraft. Because the restructure has provided the additional time and aircraft, the Army has an opportunity to significantly reduce or eliminate program concurrency and its associated risks by completing operational testing before committing funds to any production decisions.

Comanche Costs Continue to Increase

The Comanche was originally justified to the Congress as a relatively inexpensive aircraft. However, since 1985, the program has experienced significant increases in program acquisition unit cost. Funding reductions have caused the program to undergo significant restructuring, resulting in sharp decreases in planned acquisition quantities and lengthening of development schedules, thereby increasing Comanche program costs.

In 1985, the Comanche had estimated total program acquisition costs of about $61 billion for 5,023 aircraft (or $12.1 million per aircraft). In 1992, we reported that (1) as of October 1991, the program acquisition unit cost had increased to $27.4 million, (2) acquisition quantities had been reduced to 1,292 aircraft, and (3) future increases in cost per aircraft were likely.As of February 1995, the Comanche’s estimated program acquisition unit cost was $34.4 million per aircraft, a 185-percent increase from the 1985 estimate. The estimated total program acquisition cost for the planned acquisition of 1,292 aircraft is now more than $44 billion.

Both the Secretary’s decision and the Army’s restructure would extend the development program by about 3 years and, under either, increase the risk of higher total program cost and cost per aircraft. However, in reviewing the Army’s restructure proposal, DOD noted some concern over Comanche program costs for fiscal year 2002 and beyond and the large increase in investment programs projected to occur about that time. We are also concerned that the Army’s plan to defer additional developmental efforts to fiscal year 2002 and beyond may increase the risk that needed funds may not be available to perform the deferred developmental effort.

Technical Risks to Be Resolved

The Comanche program’s uncertainties in software development and aircraft maintainability increase the risk that the aircraft will not perform successfully. We believe the restructuring provides additional time to resolve these issues before the decision to enter production is made.

Software Development Is at Risk

The Comanche will be the most computerized, software-intensive Army helicopter ever built. The Army estimates that about 1.4 million lines of code are required to perform and integrate mission critical functions. With additional ground support and training software to be developed, the total program will have more than 2.7 million lines of code. This compares to about 573,000 lines of code for the upgraded Apache attack helicopter with fire control radar. The Army estimates 95 percent of the Comanche’s total software will be written in Ada, a DOD-developed programming language. The Army plans to demonstrate initial software performance with the mission equipment package, which includes the flight control system, during first flight of the Comanche, scheduled for November 1995.

The development and integration of on-board, embedded computer systems is a significant program objective. The Comanche’s performance and capability depend heavily on these systems and efforts have been ongoing to solve the problems associated with these systems. Nevertheless, (1) software development problems still exist with the Ada compilation system, (2) delays in software development and testing are occurring, and (3) improvements are needed in configuration management. If these issues are not resolved, the aircraft’s performance and capability will be degraded and first flight could be delayed.

Development Problems Remain Unresolved for the Ada Compilation System

Almost all of the Comanche software will be developed in the Ada programming language; however, software developers are not using the same version of the Ada compilation system. The Ada compilation system translates Ada code into machine language so that software can be used by the Comanche’s computers. For example, it is being used to help develop software for use on the mission equipment package that is critical for first flight.

Subcontractors and the contractor team should be using the same, qualified version of this compilation system to ensure effective software integration. However, fixes to individual compiler software problems are not being shared with all developers; therefore, they are not using a common compilation system. These problems have already delayed qualification testing of the compilation system by 1 year.

The lack of a uniform, qualified compilation system among software developers could put first flight at risk, according to the Defense Plant Representative Office. Problems with software integration may show up once integration testing begins in the June to November 1995 time frame. If that occurs, there may not be time to fix problems prior to scheduled first flight.

Delays in Software Development and Testing

The program is experiencing high turnover of software engineers at one of the contractor team’s facilities. In its December 1994, monthly assessment report, the Defense Plant Representative Office, which is responsible for contract oversight, observed that high turnover of software personnel was putting scheduled first flight at risk. Loss of key personnel has already contributed to schedule slippage in several critical software development areas. Software development for the following areas has been affected: the airborne engine monitoring system, aircraft systems management, control database, and crewstation interface management.

The contractor team has formulated a “get well” plan that is dependent on being able to hire additional personnel in these areas. However, hiring additional qualified personnel is difficult, according to the Defense Plant Representative Office, because employment would be short term.

The flight control system software verification testing is also being delayed. As of February 8, 1995, Boeing had conducted only 163 of approximately 500 tests originally planned to be completed by that date. The subcontractor responsible for developing this software has been late delivering software for testing and has provided faulty software to Boeing, according to the Defense Plant Representative Office. Boeing established a recovery plan for this area that would have resulted in a completion date in March 1995—about a 1-month delay from the original plan. However, in February 1995, the contractor revised the recovery plan to reflect a completion date of July 1995—a 5-month delay.

The flight control system is critical to first flight, according to the Defense Plant Representative Office. However, because of delays with verification testing, the Defense Plant Representative Office is concerned that the remaining verification testing, as well as, the validation and formal qualification testing will not be completed in a timely manner. As a result, first flight may be delayed. Boeing is scheduled to complete these tests prior to first flight. According to the program office, Boeing’s plan to complete the testing calls for it to be conducted concurrently. If major problems occur in any one of the testing phases, there may not be enough time to fix the problem and complete all testing before first flight.

Improvements in Configuration Management Are Needed

Configuration management is the discipline of applying technical and administrative direction and surveillance to (a) control the flow of information between organizations and activities within a project; (b) manage the ownership of, and changes to, controlled information; (c) ensure information consistency; and (d) enable product release, acceptance, and maintenance. The part of configuration management used to report software problems and changes among the contractor team and subcontractors has shortcomings that put software development at risk.

In its November 1994 monthly assessment report, the Defense Plant Representative Office observed that the lack of a common problem reporting system made proper handling of software related changes difficult. Furthermore, the report noted that this situation could adversely impact scheduled first flight of the Comanche. As of February 1995, the contractor team still did not have a common, automated database available to track problem change reports. Thus, the contractor team, as well as subcontractors, did not have visibility over changes made to software.

Key Maintainability Requirements at Risk

Maintainability requirements are important to achieving lower operating and support costs and wartime availability goals. However, these goals are at risk because key maintainability requirements such as direct maintenance man-hours per flight hour (MMH/FH), the mean time to repair (MTTR), and fault isolation may not be achievable. Individually, failure to meet these parameters may not be a significant problem; however, collectively they affect the ability of the Comanche to achieve lower operating and support cost and wartime availability objectives.

Key Maintenance Goal May Not Be Realistic

In March 1987, the Army established a 2.6 direct MMH/FH requirement for the Comanche. It represents the corrective and preventive maintenance per flight hour expected to be performed at the unit level. The Army formulated its planned wartime operating tempo for a Comanche battalion based on 6 hours a day per aircraft, or 2,200 flying hours per year. It then determined the maintenance factor needed to support this operating tempo—2.6 MMH/FH. As the MMH/FH level increases, the number of maintainers needed to sustain the 2,200 wartime flying hour goal increases, as do operating and support costs. Conversely, if the Army could not increase the number of maintainers, the planned operating tempo would have to be reduced.

The reasonableness of the Comanche’s 2.6 direct MMH/FH requirement has been debated for several years within the Army and DOD. Representatives from the program office; the Army Materiel Systems Analysis Activity, which independently evaluates program testing results; the Office of the Assistant Secretary of the Army for Research, Development, and Acquisition; and the Army Cost and Economic Analysis Center met on October 28, 1994, to discuss the direct MMH/FH goal for the Comanche program. They agreed that the 2.6-MMH/FH requirement was not a realistic, achievable goal. Consequently, Army officials reached consensus and agreed on 3.2 direct MMH/FH as the Army-wide position for this parameter. However, during these discussions, Army Materiel Systems Analysis Activity personnel noted that attaining a 3.2-MMH/FH goal represented a medium to high risk, while a 4.3-MMH/FH goal had a low to medium risk. Increasing the maintenance factor increased the number of maintainers needed and will increase estimated operating and support costs by about $800 million over a 20-year period.

The direct MMH/FH requirement does not represent the total maintenance burden for the Comanche because it does not include indirect maintenance time. The Army does not normally collect data on indirect maintenance time. According to the program office, its best estimate of indirect maintenance time, following Army guidance, is 2.5 MMH/FH, and this figure has been used for calculating manpower needs for crew chief personnel on the Comanche. Thus, the total maintenance burden assumed for the Comanche is currently 5.7 MMH/FH (3.2 direct MMH/FH plus 2.5 indirect MMH/FH).

Repair Requirement May Not Be Met

To minimize turnaround time for repairs at the unit and depot, the Army established MTTR requirements of 52 minutes for repairs at the unit level and up to 12 hours at the depot level for the Comanche. These requirements represents the average time expected to diagnose a fault, remove and repair an item, and perform an operational check and/or test flight. We determined that any increase in MTTR above 1 hour will begin to impact the Army’s wartime availability goal of 2,200 hours per year, unless additional maintenance personnel are available.

As of January 1995, the contractor team was estimating that the Army would achieve 59 minutes for unit level repairs. According to contractor team officials, the requirement was not being met because the cure time required for composite material used on the aircraft was greater than expected. The contractor team discussed changing the MTTR requirement to 1 hour; however, the program office believes the problem could be resolved and did not believe the specification should be changed. The contractor team has not yet developed MTTR estimates for depot-level repair.

Key Diagnostic System Requirement May Not Be Achieved

The Comanche’s diagnostic system is required to correctly isolate failed mechanical and electrical components at least 80 percent of the time with a high degree of accuracy. A high level of accuracy is essential as it allows maintainers to isolate and fix problems at the unit level. If the fault isolation requirement is not met, the Comanche is unlikely to achieve its MTTR requirement, thereby adversely affecting the Army’s ability to execute its maintenance concept and its wartime availability goals.

Contractor team officials stated the fault isolation requirement was very optimistic, and although they are striving to meet this requirement, it may eventually have to be changed. As of January 1995, the contractor team predicted the system could achieve an overall 69-percent fault isolation rate; however, this rate would not meet the specification for mechanical and electrical component fault isolation. There are design limitations on two components, according to the program office, and changes to bring these components into conformance with specifications would be costly and increase the weight of the aircraft. Therefore, as of January 1995, the contractor team and the program office have agreed not to take action on these components.

False Removal Rate Is High Risk

The Army established a requirement of a 1-percent false removal rate for the Comanche. A false removal occurs when a part removed from the aircraft shows no evidence of failure when tested. This requirement is dependent, to a large extent, on the success of the fault detection/isolation system in detecting and isolating failed components. Program personnel characterize the 1-percent requirement as stringent and one that will be challenging to achieve. An Army Materiel Systems Analysis Activity official believes some design improvements have occurred in this area, but the risk associated with achieving this requirement still remains high. If the Comanche does not meet this requirement, estimated operating and support costs for the Comanche will be higher than previously predicted.

The Army has not had good experience in developing fault detection/isolation and false removal systems for other aircraft. In September 1990, we reported that the fault detection and isolation system on the Apache aircraft did not always accurately detect the component that caused a particular fault, and the system detected faults that did not actually exist about 40 percent of the time. As a result, Apache maintainers had to perform additional work to locate failed components. Recently, through a reliability program, the false removal rate for the targeting and night vision systems on the Apache improved to about 10 to 15 percent, according to Army officials. This is still significantly higher than the 1-percent requirement established for the Comanche program.

Some Program Goals Are Currently Being Met

Although the program is experiencing technical problems, it is currently meeting its goals of reducing maintenance levels and keeping overall weight growth within acceptable limits for the Comanche.

The Army’s maintenance concept for the Comanche program is predicated on two levels of maintenance—unit- and depot-level maintenance. This concept is important to achieving operating and support savings predicted for the program because it eliminates the intermediate level of maintenance. Unit-level maintenance entails removing and replacing components required to return the aircraft to a serviceable condition. Depot-level maintenance requires higher level maintenance skills and sophisticated capital equipment and facilities not found at the unit level. The Army traditionally uses a three-level maintenance concept that includes intermediate-level maintenance to handle component repairs. Intermediate-level maintenance is usually located close to the battalion. It is performed on components that cannot be easily repaired at the unit level and do not require the more sophisticated repairs done at the depot level.

As of January 1995, no Comanche component had been designated for repair at the intermediate level, according to the program office. Contractor team personnel are conducting repair level analysis on Comanche components to determine whether components should be repaired at unit, intermediate, or depot facilities, according to program and contractor team officials. Any candidates identified for intermediate-level repair are reviewed for possible design changes that could allow maintenance at the unit or depot level. If economically feasible, the contractor team will make design changes to the component to preclude the need for intermediate-level repair.

As of February 7, 1995, the Comanche’s empty weight increased from its original specification of 7,500 pounds to 7,883 pounds. Although the Comanche’s weight continues to increase, it remains within the allowable design limit of 7,997 pounds. Weight increases affect vertical rate of climb performance on the Comanche. The Army established a limit of 500 feet-per-minute as the minimum acceptable vertical rate of climb performance. If the Comanche’s weight exceeds 8,231 pounds, the engine will have to be redesigned to produce enough power at 95 percent maximum rated engine power to sustain the minimum 500 feet-per-minute vertical rate of climb requirement.

Recommendation

We recommend that the Secretary of Defense require the Army to complete operational testing to validate the Comanche’s operational effectiveness and suitability before committing any funds to acquire long-lead production items or enter low-rate initial production.

Agency Comments and Our Evaluation

DOD generally concurred with the findings and original recommendations in our draft report. In commenting on the draft report, DOD offered explanations about why the problems that we identified were occurring and what they were doing to fix those problems. DOD disagreed with the report’s conclusion about false removals and stated that we had not presented any evidence that the Comanche’s 1-percent false removal rate may not be achievable. We still believe that the false removal goal is high risk and adjusted the report to more clearly reflect our concern.

Regarding our draft report recommendation that DOD develop program fixes that achieve program goals and reduce the risks we identified, DOD concurred and noted that the approved restructuring will significantly reduce risk. DOD concurred with our other draft recommendation not to commit production funds to the program until performance and mission requirements are met and noted that the program would be reviewed by DOD before approving the Army’s request to proceed to the engineering and manufacturing development phase—the Milestone II decision scheduled for October 2001.

Because DOD concurred in our draft report recommendations and is taking action on them, we are no longer including them in this report. However, our analysis of information on the restructuring obtained after we had submitted our draft report to DOD has further heightened our concerns about the risk of concurrency; therefore, we have revised the report and added a new recommendation. Under the stretched out, restructured Comanche program, operational testing is not even scheduled to begin until after the low-rate initial production decision is made. This approach continues the risks associated with making production decisions before knowing whether the aircraft will be able to perform as required.

Prior to the restructure, the Army planned to start operational testing with eight aircraft in May 2003. Under the restructured program, the Army plans to start operational testing with six helicopters by about August 2005. We believe that the stretched out time frame and the six aircraft acquired under the restructure provide sufficient time and aircraft to operationally test the Comanche prior to making any production decisions.

Additionally, because operational testing is not scheduled until about August 2005, DOD will not be in a position at Milestone II in October 2001 to adequately address whether the Comanche program is meeting its performance requirements. DOD’s comments are presented in their entirety in appendix I, along with our evaluation.

Scope and Methodology

To assess cost changes, software development, maintainability, and weight growth issues, we reviewed program documents and interviewed officials from the Department of the Army headquarters, Washington, D.C.; the Comanche Program Manager’s Office, St. Louis, Missouri; the U.S. Army Materiel Systems Analysis Activity, Aberdeen Proving Ground, Maryland; the Ada Validation Facility, Wright-Patterson Air Force Base, Ohio; and the Office of the Assistant Secretary of Defense for Program Analysis and Evaluation, Washington, D.C. We also reviewed program documents and interviewed contractor and Defense Plant Representative Office officials at the Boeing Helicopter Company, Philadelphia, Pennsylvania; the Sikorsky Aircraft Corporation, Stratford, Connecticut; and the Comanche Joint Program Office, Trumbull, Connecticut.

We conducted our review between August 1994 and February 1995 in accordance with generally accepted government auditing standards.

We are also sending copies of this report to the Chairmen and Ranking Minority Members of the Senate and House Committees on Appropriations, the Senate Committee on Governmental Affairs, and the House Committee on Government Reform and Oversight; the Director, Office of Management and Budget; and the Secretaries of Defense and the Army. We will also provide copies to others upon request.

This report was prepared under the direction of Thomas J. Schulz, Associate Director, Systems Development and Production Issues. Please contact Mr. Schulz at (202) 512-4841 if you or your staff have any questions concerning this report. Other major contributors to this report are listed in appendix II.

Comments From the Department of Defense

The following are GAO’s comments on the Department of Defense’s (DOD) letter dated April 20, 1995. 1. As DOD’s comments note, there are many measures of unit cost, such as average unit flyaway cost, program acquisition unit cost, and unit procurement cost. We believe that the program unit cost that we used in the report—which the footnote in the report defines as total research, development, and acquisition costs in current dollars—is as valid as flyaway cost to portray program cost growth over time. We have adjusted the report to more clearly define the basis of the unit cost we use. 2. These comments are dealt with on pages 11 and 12 of the report and in our responses to the specific DOD comments that follow. Report material on costs and concurrency has been revised to reflect information obtained after our fieldwork had been concluded. 3. The report does not say that maintainability goals will never be met. We pointed out that some key maintainability requirements are not being met and, therefore, there is a risk that the Army may not achieve the lower operating and support costs and wartime availability goals that it has established for this program. We also said that individually, failure to meet these parameters may not be a significant problem; however, collectively they affect the ability of the Comanche to achieve the cost and availability goals. This point is clearly illustrated in DOD’s comments on the failure of the fault isolation system. According to DOD, “Fault isolation is one of the key diagnostic system requirements. The DOD agrees that if the fault isolation requirement is not met, the Comanche is unlikely to achieve its mean-time-to-repair requirement, . . .”. 4. We still believe that this goal is very aggressive. DOD acknowledges that this goal is stringent and the Army has not had good experience in the past with false removals on other aircraft. Additionally, as noted in the report, Army Materiel Systems Analysis Activity said the risk associated with achieving this requirement remains high. We changed the section heading to emphasize the high risk.

Major Contributors to This Report

Accounting and Information Management Division

Gary L. Billen Robert D. Spence Lauri A. Bischof Michael W. Buell Karen A. Rieger The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (301) 258-4097 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Address Correction Requested