Military Readiness: DOD Needs a Clear and Defined Process for Setting Aircraft Availability Goals in the New Security Environment
Released: 2003-05-07T00:00:00

Letter

Background

DOD aircraft are used to perform a variety of different missions. However, for the purpose of this report, we have grouped them into five basic categories: (1) various models of fighter/attack aircraft, such as the F/A-18 Hornet, provide air superiority or close air support of ground forces; (2) bombers, such as the B-1 Lancer, provide long- and short-range delivery of heavy munitions; (3) electronic command and control aircraft, such as the E-3 Sentry, provide airspace and battlefield reconnaissance, command, and control services; (4) tankers and cargo aircraft, such as the KC-135 Stratotanker and the C-5 Galaxy, respectively, provide air refueling services and the ability to carry troops and equipment anywhere in the world; and (5) helicopters, with their ability to hover as well as conduct long- and short-range operations, are used for a variety of missions, including transportation of troops and equipment, air assault and reconnaissance, and search and rescue operations. Our review included a total of 49 different aircraft models (over 5,600 individual aircraft in 2002) in these five categories. These aircraft were considered by the services to be their key active-duty operational aircraft. Table 1 lists these aircraft models, along with the military service using them, and their MC and FMC goals for fiscal year 2002.

DOD Instruction 3110.5, dated September 1990, requires all military services to establish quantitative availability goals and corresponding condition status measurements for these aircraft and other mission- essential systems and equipment. The goals established must estimate the maximum aircraft performance that is achievable on the basis of the aircraft’s design characteristics and planned peacetime usage, and assuming full funding and optimal operation of the peacetime manpower and logistic support systems. Military personnel, civilian contractors, or both may perform the required maintenance under these systems. The instruction prescribes a basic set of condition status measures, including FMC, partial MC, and MC, that each service must use to describe the capability of systems or equipment. FMC indicates that an aircraft has all of the mission-essential systems and equipment it needs to perform all of its missions installed and operating safely. Mission-essential systems are those required to perform primary functions such as fire control, bombing, communications, electronic countermeasures, or radar. Partial MC indicates that an aircraft has the operable mission-essential equipment it needs to perform at least one of its missions, but not all. For example, an aircraft expected to be able to carry troops into combat during wartime in all weather conditions, as well as to be able to fly humanitarian missions during peacetime, would be considered partial MC if some of its equipment were broken and it could fly only humanitarian missions in clear weather. MC consists of the sum of the partial MC and FMC measures; that is, the number of MC aircraft is equivalent to the sum of the aircraft rated partial MC and the aircraft rated FMC. This report focuses on MC and FMC goals because the Army, Navy/Marines, and parts of the Air Force do not establish separate partial MC goals.

DOD Aircraft Experienced Widespread Problems in Meeting MC and FMC Goals

Many of DOD’s key aircraft have been unable to meet their MC and FMC goals since at least 1998. For example, during fiscal years 1998-2002, only 23-35 percent of the 49 aircraft models we reviewed were able to meet their MC goals, and 31-49 percent met their FMC goals. In most cases, the actual rates were at least 5 percentage points below the goals. Average MC and FMC rates varied by service and type of aircraft. For example, the Army and Air Force had the highest average MC rates, followed by the Marines and the Navy. These rates have increased slightly since fiscal year 2001 in all services except the Navy. Among aircraft types, the average MC rates varied from 60 to 80 percent. Average MC rates were the highest for helicopters, followed by cargo aircraft and tankers, fighter/attack aircraft, bombers, and electronic command/control aircraft. While the rates have fluctuated, MC and FMC goals have generally remained constant over time. Since 1998, only 11 of 49 aircraft models (22 percent) experienced a change to their goals—and 7 of these changes were to raise the goals.

Less Than One-Half of the Aircraft Models Met Goals

DOD’s key, high-demand aircraft have experienced widespread difficulties in meeting MC and FMC goals since at least 1998. (Appendix I provides a full listing of MC and FMC goals, rates, and other information by year for each aircraft model we reviewed.) For example, during fiscal years 1998- 2002, the percentage of aircraft models meeting their MC goals never exceeded 35 percent. (See fig. 1.) During this period, the rates for the individual aircraft models were more than 5 percentage points below their MC goals in 62 percent of the cases. The percentage of aircraft models meeting FMC goals during the same period ranged from 31 to 49 percentage points, and 71 percent of the cases were more than 5 percentage points below the goals.

At the service level, Army aircraft generally met their MC goals the most frequently, followed by the Marine Corps, Air Force, and Navy. (See fig. 2.) The same rank order held for FMC goals.

As previously shown in table 1, the level at which the goals were set showed little consistency, varying widely even among the same type of aircraft. For example, MC goals for the bombers in our review ranged from 50 to 80 percent, and MC goals for the fighters, from 65 to 83 percent.

Actual MC Rates Varied by Service and Type of Aircraft

Actual MC rates also varied between services and the various aircraft types. MC and FMC rates are based on the ratio of the number of hours an aircraft was actually available to the total number of hours it could have been available. The Navy/Marines and Air Force reduce the latter figure by the amount of time an aircraft was away for scheduled depot maintenance, while the Army does not make this adjustment. We computed the average rates by service and aircraft type from service data on the total number of hours each aircraft model was MC and FMC, and the total hours each aircraft model was available each year.

The average annual MC and FMC rates for the services as a whole are shown in figures 3 and 4. The Army and the Air Force had the highest average MC rates, at 77-83 percent during fiscal years 1998-2002; followed by the Marines, at about 71-75 percent; and the Navy, at 61-67 percent. A similar pattern follows for the average FMC rates for the services.

When grouped by type of aircraft, average annual MC rates were highest for helicopters (76-80 percent), cargo/tankers (75-79 percent), and fighter/attack aircraft (75-77 percent). Average annual MC rates for bombers (64-69 percent) and electronic command/control aircraft (60-67 percent) were somewhat lower. Average FMC rates showed similar rank orders. (See figs. 5 and 6.)

Goals Generally Remained Constant over Time

MC and FMC goals have generally remained constant over time. Since 1998, only 11 of 49 aircraft models (22 percent) experienced a change to their MC goals, FMC goals, or both. Seven models had their goals raised, and three had their goals lowered. One model’s MC goal was changed but then returned to its initial level. Ten of the 11 changes were for aircraft operated by the Air Force. The remaining change was for a Marine Corps aircraft. (See app. I for additional details.)

In fiscal year 2002, for example, the Air Force raised the MC goal for its E- 8 Joint Stars electronic command and control aircraft from 73 to 75 percent. According to officials, the E-8 is a relatively new (3-year-old) aircraft that is slowly increasing its performance level as it matures and Air Force maintenance personnel understand the aircraft better. The increase in the MC rate was based on an analysis of actual E-8 MC rates that were showing an upward trend in performance. The Air Force is the only service that routinely conducts formal reviews of its goals. Air Force officials told us that they generally try to keep the goals high because it is difficult to stop the goals from dropping further once they begin to be lowered. Moreover, officials believed that contractors need to be held to high standards to keep spare parts inventories and other aspects of maintenance at high levels. In another case, the MC goal for the Marine Corps’ F/A-18D Hornet fighter was raised from 60 to 75 percent, and its FMC goal, from 46 to 58 percent at the beginning of fiscal year 2000. According to Navy documents, this increase was due to a change in the aircraft’s assigned mission.

While most of the goals were either unchanged or increased, the Air Force’s Air Combat Command developed a set of interim goals in fiscal year 2000 for some of the fighters, bombers, and electronic command/control aircraft under its command. These interim goals were lower than its official MC goals. In 1999, the Command determined that problems with suppliers and manpower shortages were undercutting its ability to meet MC goals and lowering unit morale. To combat this problem, the Command developed the interim goals listed in table 2. In 2002, the Command returned to using the pre-2000 goals for all but six aircraft (A-10, E-3, F-15 C/D, F-15 E, RC-135, and U-2). According to Command officials, the lower goals applied only to their units. Goals for suppliers remained at official levels to keep spare parts inventories high. Neither the other services nor the Air Force’s other major commands responsible for aircraft operations have developed interim goals.

Mission Capable Problems Caused by a Combination of Factors

According to DOD officials, difficulties in meeting MC and FMC goals are caused by a complex combination of interrelated logistical and operational factors, with no dominating single problem. The complexity of aircraft design, the lack of availability and experience of maintenance personnel, aircraft age and usage patterns, shortages of spare parts, depot maintenance systems and other operational factors, and perceived funding shortages were all identified as causes of difficulties in meeting the goals. As indicated below, our work found that some indicated factors were valid causes, while the impact of others was less certain.

Aircraft Design Considered Key

Officials believe that the complexity of military aircraft affects its availability, and thus its ability to meet MC goals. Military aircraft are designed to handle a specific set of missions and provide a specific set of capabilities over a projected useful lifespan. According to officials, each aircraft can be inherently complex and maintenance intensive, or, depending upon the missions and capabilities it was designed to provide, simple and easy to maintain. For example, the B-2 bomber had the lowest MC rates (32-44 percent) of any aircraft we reviewed. However, according to Air Combat Command officials, one reason for these low rates is the complex design of the aircraft. The B-2 is a very advanced aircraft with low observable (stealthy) characteristics using new composite materials, and Air Force personnel are still learning how to maintain the aircraft. In contrast, the B-52 bomber had some of the highest MC rates (76-84 percent) of all the aircraft we reviewed. According to Air Force officials, the B-52 is a relatively simple and flexible design intended for ease of maintenance and durability.

Availability and Experience of Maintenance Personnel Have an Impact

Service officials also frequently linked shortages of the total number of maintenance personnel, as well as their experience level, to the failure to meet MC goals. Navy officials told us that the growing sophistication of their aircraft in general requires maintenance personnel to take longer to learn the complex computer and electronic skills needed to handle the aircraft. However, high demand for these skills in the private sector makes it difficult to retain personnel with these maintenance skills, leading to turnover and increasing the difficulty in meeting the MC goals. Similarly, a recent study published in the Air Force Journal of Logistics found that the number and experience level of maintenance personnel correlated highly with the MC rates of F-16 aircraft. As the number of experienced personnel assigned to an aircraft increased, the MC rates increased as well. Army officials also cited shortages of experienced maintenance personnel as a cause of lower MC and FMC rates. However, they also stated that it may be possible to raise the rates by maximizing the time that maintenance personnel actually spend maintaining the aircraft. For example, one Army Audit Agency study in 1998 found that maintenance personnel at one unit were spending about 70 percent of their time on nonmaintenance activities such as administrative duties, training, and time attending to personal duties.

Personnel management is an area that we have cited as a major management challenge and program risk for DOD. For years, DOD has been wrestling with shortages of key personnel because of retention problems. In 1999 we reported that the majority of factors cited as sources of dissatisfaction and reasons to leave the military were related to work circumstances, such as the lack of spare parts and materials needed to perform daily job requirements.

Aircraft Age and Usage Patterns Also Believed to Influence Availability

The advancing age and usage patterns of aircraft were other factors often cited by service officials as reasons why aircraft did not meet MC goals. DOD’s inventory of aircraft is getting older. The Congressional Budget Office recently reported that from 1980 to 2000, the average age of active- duty Navy aircraft rose from 11 years to more than 16 years; Air Force aircraft, from 13 to more than 20 years; and Army helicopters, from 10 to over 17 years. Logistics officials told us that aging influences on MC rates typically follow a cyclical pattern over the life of an aircraft. When aircraft are initially introduced, they go through a “shake down” period and have low MC rates as new equipment and supply systems stabilize and maintenance personnel learn to understand the aircraft. Eventually, MC rates begin to rise and then stabilize at a higher working level. However, as more and more flying time is accrued over the passing years, problems due to materials and parts fatigue, corrosion, and obsolescence increase, and MC rates begin to fall again. Modernization programs are then instituted to replace worn and obsolete equipment, and the pattern begins again.

Although age may affect MC rates, we found no statistical evidence that age alone explains difficulties in meeting MC goals. For example, our analysis of average aircraft ages and 2002 MC rates found no indication that older aircraft have the lowest MC rates. (See table 3.) With an average age of 40 years, the B-52 is the second oldest aircraft in DOD’s inventory. However, its MC rate of 81 for 2002 and historical MC rates consistently in the upper 70s and low 80s rank it among the highest performers we reviewed. According to Air Force officials at the Air Combat Command, in addition to their simplicity, B-52s have a relatively low number of actual flight hours, averaging about 16,000 hours each despite their age. These officials believed that accrued flight hours are a more appropriate measure of wear and tear than chronological age.

Moreover, according to these officials, the B-52 was originally scheduled to retire in the mid-1990s. However, because of its durability and flexibility, the Air Force decided to retain the aircraft until the average age reaches 32,000 hours, projected at about 2040.

Logistics officials also believe that MC rates are affected by usage patterns and whether the aircraft is operated under the conditions for which it was designed. Officials told us that the large increase in deployments in recent years has caused many DOD aircraft to be operated at rates higher than expected during their design, thus accelerating aging problems. For example, according to the Air Force Journal of Logistics study, F-15 fighters sent to Saudi Arabia in 1997 were flown at over three times their normal rate.

Spare Parts Inventories Critical

Shortages of spare parts have been recognized by us and others for years as a major contributor to lower-than-expected MC rates. As a result, we have also cited DOD inventory management as a major management challenge and program risk since 1990. Service officials continued to cite spare parts shortages as a frequent cause of difficulties in meeting MC goals. Spare parts shortages are caused by a number of problems, including underestimates of demand, and contracting and other problems associated with aging aircraft or small aircraft fleets.

We have reported on DOD’s problems in estimating aircraft spare parts requirements for years. For example, in 1999 and again in 2001, we reported that shortages of spare parts caused by inaccurate forecasting of inventory requirements was degrading MC rates for key Air Force aircraft such as the B-1B bomber, C-5 cargo planes, and F-16 fighters. In 2001 we reported that key Navy aircraft were also having readiness problems because of spare parts shortages resulting from underestimates of demand. Officials continued to raise this issue as an underlying factor in spare parts shortages. In addition, some officials also believed that the higher operating tempos associated with increased deployments have caused parts to fail quicker than expected, exacerbating weaknesses in forecasting inventory requirements.

Air Force officials told us that aging aircraft, in particular, may experience parts shortages and delays in repairs because original manufacturers may no longer make required parts. To obtain a new part, officials must wait for it to be manufactured. However, this may not be a high priority for the commercial supplier because of the relatively low profit potential. Alternatively, another company could make the part if the original manufacturer were willing to give up its proprietary rights. However, this can take longer and be more expensive than simply waiting for the original manufacturer. Moreover, officials also told us that spare parts inventories are sometimes reduced when aircraft are nearing the end of their projected life. For example, Air Force officials said that in the mid-1990s they began to shut down the spare parts supply for the B-52 because of its anticipated retirement. This resulted in a depletion of inventories, the canceling of contracts, and ultimately a drop in MC rates from 1997 to 2000. As a result of the decision to retain the B-52, the supply system is recovering and MC rates are moving up.

Similarly, the size of the aircraft fleet can also influence spare parts inventories and MC rates. According to officials, manufacturers may see little profit in stocking large inventories of spare parts for a small fleet of specialized military aircraft. Small fleets of aircraft can also suffer from having their MC rates strongly influenced by the MC failures of just a few aircraft. Large fleets of aircraft also have an advantage in having more opportunities to remove serviceable parts from one aircraft and install them in another—termed “cannibalizing”—thus helping to insulate their MC rates from the impact of parts shortages. However, we recently reported that while cannibalization is a widespread practice among the services, it increases maintenance personnel workloads and lowers morale and retention.

Maintenance Approach and Other Operational Factors May Affect MC Rates

Air Force and Navy officials cited changes to their maintenance approaches as a significant cause of slower repair times and lowered MC rates. In the mid-1990s the Air Force changed from a three-level maintenance approach to a two-level approach. This change moved much of the intermediate maintenance functions, such as the replacement or emergency manufacture of parts, away from the air base level to centralized maintenance depots. According to officials at both the Air Combat Command and Air Mobility Command, these changes slowed the pace of repairs significantly. Repair expertise was taken away from the base level, and aircraft were shipped away from home base more often for repairs. Moreover, officials believed that many experienced maintenance people were lost as they refused to move to other locations associated with the reorganizations. In this regard, our 1996 review of depot closures noted that DOD’s outplacement program helped limit the number of involuntary separations and that jobs were often available for employees willing to relocate.

The Army continues to use a three-level maintenance system, as does the Navy. However, Navy officials said they also changed their system in the mid-1990s by introducing the integrated maintenance concept. This approach, in contrast to the Air Force approach, increased the amount of aircraft modernization and other work performed at the base level during a time when funding for depot-level work was being reduced. However, officials believed this change overloaded the base-level maintenance systems and ultimately lowered reported MC rates.

From fiscal year 1988 to fiscal 2001, DOD reduced the number of major depots from 38 to 19. During this same period, the maintenance workforce was reduced by about 60 percent (from 156,000 to 64,500). These reductions were the result of overall force structure reductions since the end of the Cold War, as well as DOD’s desire to reduce costs by relying more on the private sector for the performance of depot maintenance. We have raised concerns that DOD’s downsizing of its depot infrastructure and workforce was done without sound strategic planning and that investments in facilities, equipment, and personnel in recent years have not been sufficient to ensure the long-term viability of the depots.

Other operational factors can also affect MC rates. For example, from 1997 to 2000, the Air Force’s B-1 bomber had a major power system problem that lowered MC rates by 12 points. To address the problem, the Air Combat Command instituted a system of frequent video teleconferences between the offices involved in the maintenance response to provide more intensive management of the response. This approach worked, as the MC rate climbed by 9 points by 2002. Management integration between the operations and logistics sides of the organization was also viewed as key. Good coordination between these two groups is essential because of the complex and multifaceted causes of MC problems. Finally, Air Force officials noted that some of the problems with Air Force MC rates could be explained by a change in reporting procedures. During the mid-1990s, the Air Force returned an aircraft to MC status after it was repaired but prior to the actual check flight to ensure that it was operating correctly. Now, the aircraft must pass the check flight before being classified as MC. Officials believe that this change would tend to lower MC rates slightly.

Funding Levels Raise Concerns

Officials from all services cited underfunding of spare parts inventories, maintenance depots, and other aspects of the maintenance and supply systems as a key problem. For example, Army and Navy officials told us that they often use remanufactured parts instead of new parts to save money.

DOD reports in its Fiscal Year 2000 Performance Report that it has increased funding for spare parts and depot maintenance requirements. For example, the report indicates that funding for depot maintenance increased from $5.58 billion to $7.01 billion from fiscal year 1997 to fiscal 1999 (most recent year that data are available). However, the report also acknowledges an unfunded requirement of about $1.18 billion in fiscal year 1999. Notwithstanding claims regarding the lack of funding for spare parts, we recently reported that when provided additional funds for spare parts, DOD was unable to confirm that those additional funds were used for that purpose.

The pressures for more funding to maintain DOD’s aircraft may well go up even more in coming years as the aircraft inventory continues to age. The Congressional Budget Office estimates that spending for operations and maintenance for aircraft increases by 1 to 3 percent for every additional year of age.

DOD’S Goal-Setting Process Is Largely Undefined and Undocumented

Despite the importance of MC and FMC goals as measures of readiness and logistical funding needs, we found widespread uncertainty over how the services’ MC and FMC goals were established and who is responsible for establishing them, as well as basic questions about the adequacy of those goals as measures of aircraft availability. The services could not explain and document how the original MC and FMC goals were set for any of the aircraft in our review. Furthermore, some officials questioned which goals are the best to use in reviewing aircraft availability: MC goals, FMC goals, or perhaps a new type of goal. DOD’s instruction provides little or no guidance on these and other key issues. DOD officials told us that the instruction has not been updated to reflect the current environment of increased deployments and other changes since the end of the Cold War.

Goals Are Important Indicators of Readiness, Operational Effectiveness, and Logistical Funding Needs

MC and FMC goals are used as fundamental measures of readiness throughout DOD, used as indicators of operational effectiveness, and used to help determine the size of spare parts inventories and other logistical resources needed to maintain aircraft availability. As a result, the level at which the goals are set can influence not only perceptions about operations and readiness, but also millions of dollars in spending for logistical operations.

In addition to the requirement to maintain MC and FMC data set forth by DOD Instruction 3110.5, the services use MC and FMC measures as a component of overall unit readiness determinations under DOD’s Global Status of Resources and Training System. The System requires commanders to rate their unit’s readiness at levels 1 (highest) through 5 on the basis of a combination of their professional judgment and the readiness ratings in four specific areas: personnel, training, equipment on hand, and equipment condition. MC and FMC measures are used to determine the ratings for equipment condition. For example, the Army measures equipment condition (termed “serviceability” by the Army) for aircraft by using the FMC rate. An FMC rate of 75 percent or more is required for a level-1 readiness rating, the highest available. Congress also requires DOD to include Status of Resources and Training System information on the condition of equipment as well as specific information on equipment that is not mission capable in its quarterly readiness reports to Congress. These reports assist Congress in its general responsibilities for overseeing DOD readiness and operations.

Similarly, according to DOD and service officials, MC and FMC goals are used as management tools within DOD units to diagnose problems and motivate personnel. For example, officials in the Air Combat Command told us that their use of lower interim goals beginning in fiscal year 2000 was an attempt to raise unit morale that had suffered as a result of their inability to meet the actual goals owing to shortages of personnel and spare parts. In this regard, DOD’s instruction specifically calls for the services to use the goals and condition status measurements, such as MC and FMC, to review maintenance and supply effectiveness and to have programs to identify and correct problems with systems and equipment.

Service officials told us that the goals also affect DOD’s funding levels because the goals are used to help determine the size of spare parts inventories and other logistical resources needed. Higher goals require more money to maintain parts inventories and other resources needed to achieve the goals. For example, officials told us that in the early 1990s, a $100 million contract for logistics support for one Air Force aircraft contained an MC goal of 90 percent. During this period, the contractor kept supply bins full of parts and MC goals were met. However, in the mid- 1990s a new contractor was brought in, and the MC goal was dropped to 85 percent. According to Air Force officials, their decision to lower the MC goal by 5 percentage points allowed the contractor to lower spare parts inventories and reduced the price of the maintenance contract by $10 million. However, MC rates also dropped and eventually fell below the new goal. The services have developed mathematical models to determine the size and cost of the spare parts inventories needed to support various levels of MC and FMC goals and other measures of aircraft availability. For example, the Navy uses a model called “Readiness Based Sparing” that takes a given FMC goal and determines the level of funding and spare parts inventories needed to reach that goal. Such models are useful in the case of spare parts inventories. However, we were not able to identify any models in widespread operational use that integrated the other influences on MC rates, such as maintenance personnel assigned, into an overall model able to predict the impact of changes in those resources on MC and FMC rates. Army and Air Force officials told us that they had recently developed such integrated models, and they are currently in limited use to test their validity. Navy officials told us that they did not yet have an integrated model.

The potential amount of funds affected by the level at which MC and FMC goals are set is large. Military service estimates of the spending of operations and maintenance funds for aircraft spares and repair parts were over $7 billion in fiscal year 2001. This figure does not include spending from other sources such as procurement and working capital funds.

Methods Used to Set Goals Unknown

Precisely how MC and FMC goals are established is unknown. DOD officials said that a combined DOD and military service team establishes operational requirements and MC goals during the acquisition process. After approval, these requirements are recorded in the Operational Requirements Document or other documents associated with the process. According to officials, part of this process involves an engineering analysis of the expected operational availability of the aircraft and the underlying level of maintenance support elements needed. “Operational availability” is an engineering term referring to the probability that equipment is not down owing to failure. In comparison, MC and FMC goals represent the expected percentage of time that an aircraft will be able to perform at least one or all of its missions, respectively.

Service officials reviewed the acquisition documents for many of the aircraft in our review, but were unable to explain and document how the actual MC and FMC goals were chosen. According to officials, many of these aircraft were acquired 20 to 30 years ago, under processes that have changed over the years, and with no clear documentation of the basis for the specific goal chosen. Moreover, there was often confusion over which organizations were responsible for setting the goals.

For example, Navy officials pointed to a 1996 Center for Naval Analyses study that attempted to determine how the MC and FMC goals for Navy aircraft were originally computed. According to the study, however, “no one knows the origin of the numbers or the method used to compute them. Now, the numbers are routed to knowledgeable people for revision, which are made without documenting the rationale for the changes.” In a July 17, 2002, letter to us, the Navy further explained that it believed that the MC goals were established in the early 1980s “to be in line with the reported status quo for the day” with “no analytical rigor applied at the time of their birth.” We requested a written explanation of how the goals were set because, despite repeated referrals to various offices over several months, no Navy official could explain how the goals were established or identify the responsible office. According to Navy officials, there was uncertainty between the program and policy offices as to who is responsible for establishing the goals and who should answer our questions.

Similarly, Army officials could not explain how their goals were set, and two separate Army organizations believed the other was responsible for setting the goals. The Army’s written response to our request for an explanation of how the goals were set (dated July 31, 2002) was prepared by officials from the Army’s Training and Doctrine Command and forwarded to us by a letter from the Office of the Deputy Chief of Staff for Logistics. The Deputy Chief of Staff’s letter states that MC goals for Army aircraft are extracted from the System Readiness Objective contained in the Operational Requirements Document established during an aircraft’s acquisition, and that the Training and Doctrine Command is responsible for establishing the System Readiness Objectives. However, the Training and Doctrine Command’s letter states that it does not set System Readiness Objectives and that the Deputy Chief of Staff for Logistics is responsible for establishing readiness goals. Nonetheless, the Training and Doctrine Command researched the operational requirements documents for the Army aircraft in our review in an attempt to answer our question about how the MC and FMC goals were set. The Command’s letter identified the operational availability requirements for most of the aircraft but did not explain how these requirements were set or make any reference to the MC or FMC goals. Officials from the Office of the Deputy Chief of Staff for Logistics told us that the Army is considering changing the FMC goals for all its aircraft to 75 percent to match the requirement for the highest-level readiness rating for equipment serviceability under the Global Status of Resources and Training System’s criterion. They did not know how the 75-percent-readiness-rating criterion was chosen.

Air Force officials also could not explain how the initial MC and FMC goals for their aircraft were established. Officials from the Air Combat Command—responsible for Air Force fighters, bombers, and electronic command/control aircraft in our review—told us that they could find no historical record of the process used to establish most of the goals. Similarly, officials from the Air Mobility Command—responsible for the cargo and tanker aircraft—stated that the Command was formed in 1992 out of elements from the Military Airlift and Strategic Air Commands and did not know how the previous Commands had established the goals. According to these officials, each of the major Commands that operate aircraft and other major weapon systems in the Air Force is responsible for establishing its own MC goals, and no one has published a standardized methodology to use. Moreover, some of the documentation related to the goals was lost when the Military Airlift and Strategic Air Commands were deactivated. Similar to the Navy, however, officials from both Commands believed that the goals were set on the basis of the historical performance of similar aircraft and/or subjective Command judgments.

While Air Force officials could not explain how the initial goals were established, they told us that their annual reviews of the goals are based on a mix of historical trend analysis and requirements reviews. The Air Force is the only service that conducts formal reviews of its goals each year. According to officials from the Air Mobility and Air Combat Commands, until 1997-98, reviews of the goals in both Commands were based on an analysis of actual historical MC and FMC rates. For example, analysts at the Air Mobility Command compared the goals with the actual rates for the previous 2 years. Depending upon actual performance, the goal could then be changed, sometimes on the basis of subjective judgments. According to Air Combat Command officials, the MC goal for the B-2 bomber was set in fiscal year 2000 using an analysis of historical rates and command judgment. The first B-2 was delivered in 1993.

In 1997-98, the two Air Force Commands began to develop so-called “requirements-based analyses” to review the standards. According to officials at the Air Combat Command, for example, it was recognized that the historical approach to reviewing the standards can perpetuate relatively low standards because it simply accepts the low funding levels and other problems that may lower MC rates without focusing on actual mission needs. The new approach attempts to factor in wartime operational requirements, peacetime flying hour requirements for pilot training, and other such requirements. A mix of both approaches is currently used by the commands to review the goals.

The services also differed in their treatment of other important aspects of managing the goals, such as whether to vary the goals on the basis of an aircraft’s deployment posture. The Navy was the only service to tier its goals on the basis of its traditional practice of cyclical deployment schedules on board its ships and aircraft carriers. Operational aircraft in the Navy follow a cyclical pattern of deploying to sea on aircraft carriers and other vessels for a set period of time, such as 6 months. Once the deployed units are replaced, they experience a stand-down period during which they recover from the rigors of deployment until it is time to begin preparing for the next movement. The Navy varies the intensity of its maintenance and its MC and FMC goals according to this pattern. Navy aircraft more than 90 days away from a deployment have goals that are 5 percentage points lower than aircraft within 90 days of a deployment, and aircraft actually deployed have goals that are 5 percentage points higher than those within 90 days of deploying. In comparison, aircraft in the Marine Corps and other services have a level approach to maintenance where the goals do not vary, and maintenance is kept at a relatively constant level. Navy officials believed that the cyclical approach to maintenance could lower overall MC rates over time compared with the level approach. This is because of the reduced maintenance attention when the aircraft are not deployed.

Adequacy of MC and FMC Goals Questioned

Some officials questioned whether the MC and FMC goals are adequate measures of an aircraft’s availability. For example, officials from the Air Force’s Air Mobility Command stated that they focused on the MC goal and not the FMC goal because their primary readiness objective is the specific mission currently assigned, not every possible mission the aircraft was designed for. Moreover, the Air Combat Command did not even establish FMC goals. This Command was the only one we reviewed that did not set FMC goals for its aircraft. Air Combat Command officials told us that they could find no documentation to explain why the Command did not establish FMC goals.

In contrast, Army officials stated that their units focus primarily on the FMC goal because it is directly connected to readiness ratings under the Status of Resources and Training System. Furthermore, Navy officials stated that the military is moving away from the MC and FMC goals in newer aircraft, such as the Joint Strike Fighter. This is because the MC and FMC goals provide only a limited historical perspective and do not address issues that are important to war-fighting commanders such as how often an aircraft can fly missions over the course of a day and the probability that the aircraft will complete its mission. The Joint Strike Fighter, for example, is using a concept called “mission reliability” instead of MC and FMC goals. Mission reliability is the probability that the Joint Strike Fighter will complete its required operational mission without a failure. According to Navy officials, the predictive value and information on flight frequency and reliability provided by this new measure is very valuable to war-fighting commanders and is better for mission-planning purposes than the MC and FMC measures. Officials said that the mission-reliability concept could be used throughout DOD’s inventory of aircraft.

DOD Instruction Provides Little or No Guidance on Key Issues

DOD Instruction 3110.5 provides only vague or no guidance on many of the key issues raised in this report. For example, the instruction requires each military service to establish availability goals for its mission-essential systems and equipment, and a corresponding set of condition status measures relative to those goals. The instruction specifically identifies MC, FMC, and other specific capabilities as measures that the services must maintain. However, it does not identify the specific goals that must be established—MC, FMC, or any other—or the primary readiness objective to be served. In this regard, the instruction states that the services should assume planned peacetime usage in setting the goals. According to Air Force officials, peacetime usage can be more taxing than wartime usage because of the extra training and other requirements. Air Combat Command officials told us that they believed that the instruction regarding what goals—including the FMC goal—were required to be established was unclear.

The instruction also provides little guidance on the methodology to be used in setting the goals. It states that the services should provide estimates of the maximum performance that is achievable, given the design characteristics of the aircraft, and that full funding and optimal operation of the logistics support system should be assumed. Service officials said they believe that actual levels of funding, personnel, spare parts inventories, and other key resources should be factored into the process of setting the goals, since full funding has not been provided for years. The instruction is silent on the issue of whether it is appropriate to use historical trends of similar aircraft in determining the goals, as opposed to a more analytical approach using actual requirements, for example. The instruction is also silent on whether the aircraft availability goals should vary on the basis of the aircraft’s deployment posture. Moreover, it includes no requirement for the services to identify the readiness and cost implications of setting the goals at different levels, to help clarify the pros and cons of available choices and the guiding principles used to decide on those choices.

Similarly, the instruction provides little organizational structure for the goal-setting process in DOD. For example, it does not require the services to identify one office as the coordinating organization for goal-setting and other related activities. Furthermore, it does not require the services to document the basis for the goals chosen or outline any of the basic historical documentation that should be maintained for goal-setting and other key activities during the process.

According to DOD officials from the office responsible for the instruction, DOD Instruction 3110.5 dates back to the 1970s when readiness concerns had reached a high point. The focus was on getting the services to set benchmark readiness goals, and the instruction gave them latitude to choose those goals, the methods for setting them, and the processes for managing them. The instruction was revised in 1990. However, officials told us that it has not been updated to reflect the current environment of frequent deployments and other changes since the end of the Cold War, and some now consider it a relic.

We performed our work from February through November 2002 in accordance with generally accepted government auditing standards. The final publication of this report was delayed by the impact on DOD’s report review and classification process of the terrorist attacks on September 11, 2001 and DOD’s preparations for potential conflict in Iraq.

Conclusions

While many of DOD’s key aircraft are not meeting MC and FMC goals, it is difficult to determine how significant this problem is because of the uncertainty and lack of documentation of the basis for the existing goals. Moreover, without knowing the basis for the existing goals, it is also difficult to know whether that basis is appropriate for the demands of the new defense strategy.

DOD’s Instruction 3110.5 fails to clearly define the specific availability goals that all services must establish. Without the perspective provided by clear, consistent, and up-to-date goals, the perceptions of actual performance are subject to continuing uncertainty and disagreement, and confidence in the funding requests based on those perceptions is undermined. Moreover, the lack of a standard methodology for the services to use in setting the goals removes a safeguard for objectivity from the process, risking the possibility that the methods used do not realistically reflect actual requirements. This risk is increased when there is uncertainty or disagreement over basic questions such as whether it is appropriate to base the goals on a historical analysis or an analysis of actual requirements, and whether full funding of logistical support systems should be assumed in an era of reduced funding. Furthermore, the absence of information on the readiness and cost implications of setting the goals at different levels results in a lack of understanding of the pros and cons of available choices and the guiding principles used to make those decisions. Ultimately, inappropriately set goals can unnecessarily raise or lower the cost of spare parts inventories and other logistical resources by millions of dollars.

Also, DOD’s instruction requires the services neither to designate one office to coordinate the establishment and maintenance of aircraft availability goals, nor to document the basis for the goals chosen or other key issues in the process. Clear responsibilities and requirements in these areas are fundamental to the effective management of any performance system. Without the transparency provided by adequate documentation of the process, neither DOD nor the Congress can be reasonably assured that the services have selected the optimal goals on the basis of preferred principles.

Recommendations for Executive Action

To ensure that aircraft availability goals and their performance measures are appropriate to the new defense strategy and based on a clear and defined process, we recommend that (1) DOD and the services determine whether different types of aircraft availability goals are needed, (2) as appropriate, DOD and the services validate the basis for the existing MC and FMC goals, and (3) the Secretary of Defense revise DOD Instruction 3110.5 to clearly define the specific aircraft availability goals required to be established by the military services and their accompanying performance measures; establish a standard methodology identifying objective principles of analysis to be used by all services in setting the goals, including an identification of the readiness and cost implications of setting the goals at different levels; and require each service to identify one office to act as a focal point for coordinating the development of the goals and for maintaining a documentary record of the basis for the goals chosen and other key decisions in the goal-setting process.

Agency Comments and Our Evaluation

In written comments on a draft of this report, DOD concurred or partially concurred with all our recommendations. The department agreed to determine whether different types of aircraft availability goals are needed, including the option of tailoring such goals to unique military service and mission requirements. DOD also agreed to validate the basis for the existing goals, including the DOD Instruction 3110.5 requirement that full funding of support systems be assumed in establishing availability goals. In addition, DOD indicated that it would explore alternative methodologies for setting goals, such as one based on unit deployment cycles currently in use by the Navy.

DOD partially concurred with our recommendation for a series of revisions to DOD Instruction 3110.5. It agreed with our recommendation that the instruction be revised to require each service to designate a focal point for the development and historical documentation of the goal-setting process. However, DOD did not agree with the part of our recommendation calling for it to include the performance measures associated with the aircraft availability goals in the instruction. DOD believed that that requirement implied that those performance measures should be the sole or primary measure of the overall state of materiel readiness. That was not our intent. Our recommendation is meant to ensure that the goals and accompanying performance/status measures selected are clearly defined in the instruction. As pointed out in the report, this is not currently the case. We agree that determinations of overall materiel readiness require the consideration of a variety of factors, such as maintenance manning and supply fill rates, as well as metrics such as an aircraft’s availability. However, we believe that the instruction should continue its current requirement to include performance/condition status measures relative to those goals. Clearly identifying the goals that are sought and their performance measures in the instruction will help avoid further uncertainty and disagreement over the level of basic aircraft performance, and does not preclude the consideration of other metrics in broader assessments of overall readiness. For these reasons, we believe no change to our recommendation is needed.

DOD also disagreed with the part of our recommendation calling for the Secretary of Defense to revise the instruction to establish a standard methodology identifying objective principles of analysis to be used in setting the goals. It believed that the services should establish the detailed analytical methodology because the types of goals and their basis may vary by service, and the services have a better understanding of the differences and complexities of their individual environments. We agree with the need for some leeway at the service level to handle individual differences between them. However, we continue to believe that all services should adhere to a standardized set of overarching principles of analysis in order to safeguard objectivity and transparency in the goal setting process. Such principles could be identified in coordination with the services during the department’s planned evaluation of the basis for the current goals and alternative methodologies. The services could then develop detailed methodologies consistent with these principles but tailored to their individual environments. Consequently, no change to our recommendation is required.

The department’s comments are reprinted in appendix III. DOD also provided technical comments, which we incorporated as appropriate.

As arranged with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this letter. At that time, we will send copies to the appropriate congressional committees; the Secretaries of Defense, the Army, the Navy, and the Air Force; the Commandant of the Marine Corps; and the Director, Office of Management and Budget. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

Please contact me at (757) 552-8100 if you or your staff have any questions concerning this report. The major contributors to this report are listed in appendix IV.

Appendix I: Mission Capable Goals and Rates, Fiscal Years 1991-2002

Legend: MC = mission capable, FMC = fully mission capable, FY= fiscal year, EC/C = electronic command and control.

Fiscal year 2002 rates are as of February for the Army, Navy, and Marine Corps, and March 31 for the Air Force.

Aircraft ages are as of September 30, 2001 for the Navy/Marines; December 31, 2001, for the Air Force; and April 2002 for the Army.

Aircraft costs/flying hour are as of January 2001 for the Army, and September 30, 2001, for the Air Force, Navy, and Marines.

Appendix II: Scope and Methodology

To identify Department of Defense (DOD) and service policies and practices regarding mission capable (MC) goals and rates, we obtained briefings, reviewed DOD and service regulations and prior reports by us and others; and interviewed officials at the Office of the Secretary of Defense: the Office of the Joint Chiefs of Staff; headquarters offices of the Army, Navy/Marine Corps, and Air Force; and aviation commands and other locations as appropriate.

To determine whether key DOD aircraft were meeting established MC and FMC goals, we requested that each service identify its key active-duty operational aircraft. We excluded reserve units from the scope of our review, as well as active-duty training units and executive aircraft used to transport officials on official business. The resultant list included a total of 46 different models of aircraft from the four military services, which we categorized into five basic types: bombers, cargo/tanker aircraft, electronic command/control aircraft, fighter/attack aircraft, and helicopters. Three aircraft models (F/A-18A, F/A-18C, and EA-6B) were used by both the Navy and Marine Corps. For our review, we counted the Navy and Marine Corps versions of these aircraft as separate models, resulting in a total of 49 aircraft models for review. We requested MC and full mission capable (FMC) goal and rate data, aircraft age and cost, and other data for these aircraft back to 1991 to provide a historical perspective on goal and rate history. The Army and Air Force provided comprehensive data from fiscal year 1991 to mid-fiscal year 2002. However, the Navy and Marine Corps could provide data separated by service only from fiscal year 1998 forward. These services changed their reporting system in 1998 and were unable to provide comparable data for prior years. As a result, we focused our report on the 5-year period beginning in fiscal year 1998. However, we included the full array of Army and Air Force data in appendix I. We used these data to conduct analyses of whether the aircraft were meeting their goals. We also provided each service with these databases for review, and they confirmed the results for accuracy.

To identify the causes of difficulties in meeting MC and FMC goals, we reviewed prior reports by us and others and conducted a variety of comparative analyses of our data by service, aircraft type, model, age, cost, and fiscal year. We then held discussions with each service to gain their perspectives on the causes of observed difficulties in meeting the goals.

To determine whether DOD has a clear and defined process for setting MC and FMC goals, we reviewed DOD Instruction 3110.5 and other regulations and conducted discussions with officials from the Office of the Secretary of Defense and service headquarters in Washington, D.C., and with officials from the headquarters of the Air Force’s Air Mobility and Air Combat Commands; the Naval Air Systems Command; and Army Training and Doctrine Command officials at Fort Rucker, Alabama. Because of the difficulty in obtaining clear information on this issue, we also wrote formal letters of inquiry to the Secretaries of the Army and Navy requesting clarification of how the goals were established. Their responses to those letters of inquiry were used in preparing our report.

We performed our work from February through November 2002 in accordance with generally accepted government auditing standards. The final publication of this report was delayed by the impact on DOD’s report review and classification process of the terrorist attacks on September 11, 2001, and DOD’s preparations for potential conflict in Iraq.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to those named above, Bernice Benta, Katherine Chenault, and R.K. Wild made key contributions to this report.