C-17 Aircraft: RM&amp;A Evaluation Less Demanding Than Initially Planned
Published: 1996-07-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed the Air Force's reliability, maintainability, and availability (RM&A) evaluation of the C-17 aircraft, focusing on: (1) RM&A planning, preparation, execution, and results; and (2) whether the evaluation demonstrated the aircraft's wartime surge rate.
section_title: What GAO Found
paragraphs: GAO found that: (1) the Air Force reported that the C-17 aircraft met or exceeded 10 of the 11 contract requirements during its RM&A evaluation, but the evaluation was less demanding than originally planned; (2) although the revised RM&A evaluation plan increased total flying hours, the number of sorties, and average wartime sortie duration, it decreased the ratio of sorties to flying hours, which weakened the application of RM&A measurement criteria and lessened the stress on the aircraft; (3) the RM&A evaluation also had fewer airdrops and austere airfield landings than originally planned, and the aircraft flew cargo loads that averaged less than one-half the weight projected in contract specifications; (4) three years of operational testing show that the aircraft generally met RM&A requirements with the exception of those related to built-in-test parameters; (5) the RM&A evaluation was not a statistically valid test for determining C-17 fleet wartime utilization rates because the test's duration was too short; and (6) the incentive fee should have been reduced, since the aircraft could not perform the formation personnel airdrop mission under operational conditions, or the aeromedical evacuation mission.

section_title: Letter
paragraphs: In response to your request, we reviewed the Air Force’s reliability, maintainability, and availability (RM&A) evaluation of the C-17 airlifter. Specifically, we reviewed the planning and preparation for the evaluation, monitored its execution, and assessed the results reported by the Air Force. In addition, as you requested, we obtained information on the extent that the evaluation would demonstrate the C-17’s wartime surge rate.
subsections: section_title: Background
paragraphs: The C-17 military transport, which is being produced for the Air Force by the McDonnell Douglas Corporation, is designed to airlift substantial payloads over long ranges without refueling. The Air Force intends the C-17 to be its core airlifter and the cornerstone of its future airlift force. The Congress had appropriated about $20.7 billion and authorized the acquisition of 40 aircraft, through fiscal year 1996, for the C-17 program. The $20.7 billion includes $5.9 billion for research and development,  $14.6 billion for procurement, and $170 million for military construction. The Congress has also authorized the Department of Defense (DOD) to enter into a multiyear contract for the acquisition of the remaining 80 aircraft of the 120 aircraft C-17 program. As of July 3, 1996, 27 aircraft have been delivered.
The C-17 development contract required the Air Force to conduct a 30-day evaluation of the aircraft’s compliance with RM&A specifications. The evaluation was also used to determine how much of a $12-million incentive fee the contractor was entitled to for meeting those specifications. In October 1992, the Air Force developed a draft RM&A evaluation plan that was closely tailored to the contract specifications. The plan was revised during 1994 and issued in July 1994.
The 30-day RM&A evaluation was conducted between July 7 and August 5, 1995. It consisted of a 23-day peacetime segment and a 7-day wartime segment. Aircraft operations, using 12 aircraft, were conducted at  6 U.S. airfields and 1 overseas base. Table 1 shows the number of missions, sorties, and flight hours flown during the evaluation.
Missions included logistics (transporting equipment, personnel, and supplies); joint operations (training with equipment and personnel from the Army); and peacetime aircrew training. The wartime logistics missions were designed to simulate long-range movement of equipment, personnel, and supplies to forward operating bases or small austere airfields. Peacetime and wartime missions included aerial refueling; equipment and personnel airdrops; formation flying; low-level operations; and operations into small, austere airfields. The wartime missions ranged from 12.5 to  26 hours, while the peacetime missions ranged from 2 to 20.5 hours.
By the end of the evaluation, the C-17 fleet had logged about 13,000 total operational flying hours since initial squadron operations began in 1993. The RM&A evaluation represents about 2 percent of the 100,000 flying hours needed to meet aircraft fleet maturity.
section_title: RM&A Evaluation Was Less Demanding Than Originally Planned
paragraphs: The C-17 met or exceeded 10 of the 11 RM&A evaluation contract specification requirements. (See app. I.) However, the RM&A evaluation was less demanding than originally called for in the contract specifications and the 1992 draft RM&A plan. The RM&A evaluation, based on the 1994 revised plan, decreased the ratio of sorties to total flying hours. The decrease weakened the link between the evaluation as executed and the RM&A measurement criteria. In addition, the evaluation was less demanding because the number of airdrops and landings on small austere airfields was decreased and lighter average cargo loads than called for in the contract specifications were carried.
subsections: section_title: Revised Mission Profiles Reduce the Ratio of Sorties to Flying Hours
paragraphs: The 1992 draft RM&A evaluation plan was based on 25 C-17 mission profiles representing the aircraft’s projected peacetime and wartime usage over a 30,000-hour airframe life included in the development contract. In developing its 1992 draft plan, the Air Force conducted extensive analyses and reviews to ensure that the plan adhered to the contractual requirements.
In January 1994, as part of the settlement related to the C-17 development program between DOD and the contractor, DOD directed the Air Force and the contractor to revise the C-17 RM&A evaluation plan to make it more operationally realistic. That is, to more realistically mirror the planned use of the aircraft. In addition, because of reliability problems with the C-17, the scheduled November 1995 Defense Acquisition Board was to consider the evaluation results when it decided whether to continue the C-17 program beyond 40 aircraft. As part of the 1994 revisions, the Air Mobility Command changed the mission profiles used in the October 1992 draft plan because they did not represent complete and comprehensive missions. Command officials were also concerned that the 1992 draft plan would not demonstrate the aircraft’s wartime surge utilization rates included in the C-17 Operational Requirements Document.
In July 1994, the Air Force issued the revised RM&A evaluation plan. The plan included a wartime scenario representative of a major regional contingency, additional sorties to simulate complete missions, and additional flying hours to increase the aircraft’s utilization rate.
The revised mission profiles in the final RM&A plan increased the total number of flying hours, number of aircraft sorties, and average wartime sortie duration, but did not maintain the proportional mix of sorties to flying hours that was based on contract specifications. The impact of these changes was longer duration wartime sorties and a reduced ratio of sorties to flying hours, resulting in less stress on the RM&A aircraft than originally planned. Longer missions with fewer cycles, such as strategic intertheater missions, place less stress on an aircraft and will result in longer aircraft life. The 1992 draft evaluation plan provided for 1,725 total flying hours. The RM&A evaluation increased the level to 2,259 flying hours, an increase of 31 percent over the draft plan. The total number of sorties flown increased 12 percent, but the average sortie time increased 17 percent. Peacetime sorties increased 34 percent, from 248 to 334, but the number of wartime sorties decreased 15 percent, from 211 to 179. Although the change in the duration of the average peacetime sortie was negligible, the average wartime sortie increased by 50 percent, from 3.99 to 5.97 hours. (See app. II.)
Because the average wartime sortie significantly increased, the number of sorties in relation to the number of flying hours was less than planned in the 1992 draft RM&A evaluation plan. We estimate that if the average duration of peacetime and wartime aircraft sorties had not changed, the Air Force would have needed to fly 90 additional sorties. This represents a 15-percent increase in the number of sorties necessary to maintain the proportional mix of flying hours to aircraft sorties identified in the 1992 draft evaluation plan. (See app. III.)
section_title: Changes to Mission Profiles Weaken Application of RM&A Measurement Criteria
paragraphs: The ratio of flying hours to sorties specified in the contract and 1992 draft plan mission profiles was based on the profiles used in the development of selected RM&A measurement criteria. In addition to reducing the stress on the RM&A aircraft, changes to the original mission profiles weakened the link between the RM&A evaluation scenarios and the assessment criteria developed using the original profiles.
The Air Force used the C-17 lifetime mission profiles in the contract specifications to develop the test profiles in the 1992 draft of the RM&A evaluation plan. These lifetime mission profiles were also used to develop a number of the C-17 RM&A growth curve parameters, such as mission completion success probability, full mission capable rate, and partial mission capable rate. The RM&A growth curves, based on total C-17 fleet flying hours, are the criteria used to measure the C-17 RM&A results.
A 1981 report by the contractor noted that the operational profiles flown during the RM&A evaluation must be the same as the profiles used to develop the growth curves. Since the original mission profiles were used as a basis for developing RM&A growth curve criteria, a revision in the profiles required a corresponding adjustment in the respective growth curves. The failure to make such an adjustment affected the use of the growth curves as RM&A measurement criteria.
section_title: Airdrops and Austere Airfield Landings Were Reduced
paragraphs: The total number of airdrops and austere airfield landings accomplished in the RM&A evaluation were less than called for in the 1992 draft plan, thus causing less stress and wear on the C-17 aircraft and its subsystems.
The total number of airdrops was reduced from 189 to 158, a 16-percent reduction. Wartime airdrops were decreased by 92 percent, from 50 to 4. Air Force officials stated that they significantly decreased the number of wartime airdrops because the 1992 Mobility Requirements Study and the 1995 Mobility Requirements Study Bottom-Up Review Update did not include airdrop as a requirement for a major regional contingency warfighting scenario.
The number of C-17 small, austere airfield landings was 16 percent less than called for in the 1992 draft plan—138 instead of 164. According to Air Mobility Command officials, they reduced the number of landings from 164 to 148 because they did not believe the additional landings were needed to determine the RM&A evaluation impact and an additional  10 planned landings were not accomplished due to mechanical or environmental problems.
section_title: Average Cargo Loads Less Than Planned
paragraphs: Although the type of cargo carried during the RM&A evaluation was realistic, the average weight of the loads was less than half that projected in the mission profiles in the contract specifications. As a result, the aircraft and its subsystems experienced less stress and wear during the evaluation.
Based on the mission profiles in the contract specifications, the average cargo weight per mission over the lifetime of the C-17 aircraft is 48,649 pounds. However, the aircraft only carried an average cargo weight of approximately 23,000 pounds during the RM&A evaluation. In addition, the actual average cargo weight carried during landings on small austere airfields was nearly 2.5 times less than the average cargo loads projected in the contract specifications (about 18,600 rather than 45,000 pounds). We are currently reviewing the C-17’s performance in Bosnia. This work should provide greater insights into aircraft performance when carrying heavier loads.
section_title: Extent to Which Utilization Rates Were Demonstrated
paragraphs: One reason for revising the 1992 draft RM&A evaluation plan was to demonstrate the wartime surge utilization rate included in the C-17 Operational Requirements Document—that is, operate 15.2 flying hours a day per aircraft for 45 days. Aircraft utilization rate goals were met and slightly exceeded during the RM&A evaluation. However, the evaluation was not intended to provide a statistically valid basis for predicting the C-17’s ability to meet its wartime surge rate. It did not demonstrate what a mature C-17 fleet would do during 45 days of wartime surge operations. The evaluation simply demonstrated that high utilization rates could be achieved over a 48-hour period.
The actual peacetime utilization rate was 4.3 hours per aircraft. The wartime sustained rate was 12.7 hours, with wartime surge rates of 16.6 and 17.1 hours demonstrated during two 24-hour periods. According to DOD and Air Force officials, it would not be economically feasible to conduct more realistic tests because of the large amount of flying hours and resources required.
Moreover, while utilization rates are used as one basis for budgeting for logistics resources and mission planning, a higher utilization rate does not necessarily mean that one aircraft is a better airlifter than another. Simply stated, utilization rate is the number of hours, per aircraft, that a fleet of airplanes is in the air on a given day. More time in the air yields higher utilization rates, more time spent on the ground yields lower utilization rates. The rate is a function of the total airlift system that includes, among other things, aircraft, personnel, airfields, logistics resources, and concepts of operation. All these factors influence the attainment of a utilization rate objective, and most have little or nothing to do with an aircraft’s inherent capability. For example, utilization rates can be increased by longer mission flying times, slower airspeeds, aircrew augmentation, and ramp space availability. Conversely, a faster aircraft flying the same distance will have a lower utilization rate.
section_title: Incentive Fee Award Overstated Given Equipment and Capability Exclusions
paragraphs: The Air Force awarded the C-17 contractor $5.91 million of the maximum $12-million incentive fee. However, our review showed that amount was $750,000 more than justified under the contract. The amount should have been reduced because the C-17 aircraft were not full mission capable during the evaluation. (See app. IV for our calculation of the appropriate incentive fee.)
subsections: section_title: Aircraft Could Not Perform Formation Personnel Airdrop or Aeromedical Evacuation Missions
paragraphs: According to the C-17 development contract, the RM&A incentive fee was to be based on the degree that the contractor met each of 11 individual RM&A parameter goals. That is, to receive the total $12-million payment, the contractor had to achieve the goals for each of the 11 parameters. If any parameter was not met, the payment was reduced by the amount for that parameter and half of the amounts for the remaining parameters. The contractor was awarded only $5.91 million because the C-17 did not meet the requirement for the built-in-test false indication parameter.
In awarding the $5.91-million fee, the Air Force gave the contractor credit for meeting the full mission capable goal. In our opinion, none of the aircraft should have been considered full mission capable during the evaluation. First, the Air Force, based on the results of developmental testing, had restricted the aircraft from executing the formation personnel airdrop mission under operational conditions for safety reasons. This mission was a requirement identified in C-17 operational documents. The restriction on formation personnel airdrop existed because turbulence caused by the aircraft can cause injuries to paratroopers. As a result, the aircraft are not permitted to fly in sufficiently close formation to airdrop the required number of personnel under operationally representative conditions as required by the contract specification.
Second, the aircraft were not considered effective for the aeromedical evacuation mission, which was not completely tested during the RM&A evaluation. The aircraft were reconfigured to demonstrate this capability, but not all the systems that would be needed to accomplish the mission were used. Initial operational test and evaluation testing, which included the information developed during the RM&A evaluation, identified a number of problems that prevented the aircraft from being considered able to perform the aeromedical evacuation mission. For example, the emergency oxygen supply to patient litters was defective. As a result, the Army has classified the aircraft as not functionally effective for aeromedical evacuation.
section_title: Recommendation
paragraphs: We recommend that the Secretary of Defense direct the Secretary of the Air Force to initiate action to recover the $750,000 in incentive fee overpayment from the contractor.
section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on a draft of this report, DOD partially concurred with our findings but did not concur with our recommendation. DOD stated that the 1994 plan was actually more extensive and more operationally representative than the draft 1992 plan because it increased the total flying hours, the number of sorties, wartime sortie duration, aerial refueling, and formation flying missions. However, DOD acknowledged that the 1994 plan reflected (1) a 30-percent reduction in the number of airdrop sorties, (2) a 10-percent reduction in the number of small austere airfield landings, and (3) more than a 50-percent reduction in the average cargo loads carried during the evaluation compared to the 1992 draft plan. DOD indicated that the 1992 draft plan should not be used as a benchmark, rather the contract specification, including the 1994 Settlement Agreement between DOD and the contractor, should have been used. Further, DOD stated that not adjusting the growth curves to account for the changes made in the plan would have had only minimal impact on the results of the evaluation.
We did not use the 1992 draft plan as a benchmark. Rather, we pointed out that scenarios in the 1992 draft plan and the growth curves, which are the criteria used to measure the success of the evaluation, were both based on the same factors from the contract specification. The 1994 plan changed the scenarios being flown in the evaluation to make it more operationally realistic and to more closely resemble a major regional contingency. However, the growth curves were not adjusted. DOD provided no documentation to support its assertion that adjusting the growth curves would have had only minimal impact. Moreover, the C-17 contractor has stressed that the profiles flown during the evaluation must be the same as those used to develop the growth curves.
DOD acknowledged that the limited wartime surge activities during the RM&A evaluation did not provide a statistical basis for predicting the C-17’s ability to meet its wartime surge rate. DOD said we questioned the value of utilization rates in this report, even though in a prior report we had indicated that utilization rates were a useful statistic when comparing aircraft. Our point in the prior report was that the value of comparing utilization rates was undermined when DOD artificially constrained the utilization rate of one aircraft while using the planned wartime surge utilization rate for another. However, to assure that our position in this report is clear, we have modified the text dealing with utilization rates.
DOD disagreed with our recommendation to seek reimbursement of $750,000 from the contractor, asserting that the aircraft was properly considered full mission capable as long as all the equipment required for the mission was available and operative. The contract specification defines full mission capable as the aircraft being capable of performing all of its design missions. Since it could not perform the formation personnel airdrop and aeromedical evacuation missions, we believe that the aircraft was incorrectly listed as full mission capable. The aircraft is restricted from performing the formation personnel airdrop mission for safety reasons. While the aircraft were reconfigured to perform the aeromedical evacuation mission, some of the equipment was not tested to ensure it was operating as needed to enable the aircraft to perform the mission. Further, the aircraft was classified as not functionally effective for aeromedical evacuation as a result of initial operational test and evaluation testing because of a number of problems, including equipment problems.
We, therefore, continue to believe that the aircraft should not have been considered as full mission capable and the contractor should not have been paid the incentive award fee of $750,000 for meeting the full mission capable objective.
section_title: Scope and Methodology
paragraphs: To determine the overall performance of the C-17 during the evaluation, we monitored the conduct and coordination of the RM&A evaluation from the 437th Airlift Wing, Charleston Air Force Base, South Carolina. This included the daily RM&A evaluation activities of the exercise as well as related data collection and documentation activities. We also flew on selected C-17 missions and observed ground operations at C-17 operating bases, including North Auxiliary Airfield, South Carolina; Pope Air Force Base, North Carolina; and forward operating bases at Barstow-Daggett Municipal Airport, California, and Bicycle Lake Army Airfield, California.
To determine the validity of the test design, mission mix, and operational realism of the exercise, we analyzed the RM&A evaluation plan. Specifically, we reviewed its purpose, structure, preparation, and execution as well as the results of the evaluation. We also interviewed officials from the 14th Airlift Squadron; the 17th Airlift Squadron; the 437th Airlift Wing; the 315th Reserve Airlift Wing; Air Mobility Command Headquarters; the C-17 System Program Office; C-17 Site Activation Task Force; San Antonio Air Logistics Center; Air Force Operational Test and Evaluation Center; Air Force Office of Operational Test and Evaluation; Headquarters U.S. Air Force; U.S. Army Test and Evaluation Command; U.S. Army Test and Experimentation Command; Institute for Defense Analysis; and McDonnell Douglas, the C-17 contractor.
We conducted our review from June 1995 to March 1996 in accordance with generally accepted government auditing standards.
We are sending copies of this report to the Chairmen and Ranking Minority Members of the Senate Committee on Armed Services; the Subcommittee on Defense, Senate Committee on Appropriations; the House Committee on National Security; the Subcommittee on National Security, House Committee on Appropriations; the Secretaries of Defense and the Air Force; and the Director of the Office of Management and Budget. We will also provide copies to other interested parties as requested.
If you or your staff have any questions concerning this report, please contact me on (202) 512-4841. The major contributors to this report are listed in appendix VI.
section_title: Comparison of C-17 30-Day RM&A Evaluation With Results of Operational Test and Evaluation
paragraphs: RM&A evaluation July 7 to August 5, 1995 MTBM(i)
MTBM(c)
Mission capable (capable to perform at least one mission) Full mission capable (capable to perform all missions) Mission completion success probability (complete mission objectives without experiencing failure or performance degradation due to equipment problems) Mean time between maintenance-inherent (mean flight hours between unscheduled, on-equipment, inherent maintenance actions) Mean time between maintenance-corrective (mean flight hours between unscheduled corrective actions) Mean time between removal (mean flying hours between removal of any repairable equipment) Maintenance man hours per flying hour (total maintenance hours expended for each flight hour) Mean man hours to repair (the mean maintenance man hours required to complete a corrective maintenance action) Built-in-test fault detection (percentage of occurrences in which BIT correctly detects a malfunction) Built-in-test fault isolation (percentage of occurrences in which BIT correctly isolates a detected malfunction to the failed equipment item) Built-in-test false fault indication (percentage of occurrences in which BIT indicated a malfunction when none existed)
section_title: Comparision of Flight Hours, Sorties, and Average Sortie Time Between Original RM&A Plan, Revised Plan, and Actual
paragraphs: Revised plan (July 1994)
Original plan (October 1992)
Difference between revised and original (36) (17) (32) (15) (0.16) (05)
section_title: Comparison of Actual RM&A C-17 Aircraft Cycles/Sorties With Estimated Cycles/Sorties Using Average Sortie Times Contained in Original Plan
section_title: RM&A Evaluation Incentive Award Fee Calculation
paragraphs: MTBM (I)
MTBM (C)
section_title: Comments From the Department of Defense
section_title: Major Contributors to This Report
subsections: section_title: National Security and International Affairs Division, Washington, D.C.
section_title: Los Angeles Field Office
paragraphs: Noel J. Lance Dorian R. Dunbar Larry J. Bridges The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.