Defense Acquisitions: Issues Concerning Airlift and Tanker Programs
Released: 2007-03-07T00:00:00

Letter

Background

DOD must be capable of rapidly deploying armed forces to respond to contingency and humanitarian operations around the world. Airlift and tanker aircraft play a vital role in providing this capability. Over the past 25 years, DOD has invested almost $141 billion to develop, procure, and modify its airlift and tanker forces with an additional investment planned for fiscal years 2007 through 2011 of $32 billion. Recent annual funding levels are at the highest levels in two decades. (See figure 1.)

In December 2005, DOD issued a report on the study of its mobility capabilities. The goal of this Mobility Capabilities Study was to identify and quantify the mobility capabilities needed to support U.S. strategic objectives into the next decade. The MCS determined that the projected mobility capabilities are adequate to achieve U.S. objectives with an acceptable level of risk during the period from fiscal years 2007 through 2013; that is, the current U.S. inventory of aircraft, ships, prepositioned assets, and other capabilities are sufficient, in conjunction with host nation support. The MCS emphasized that continued investment in the mobility system, in line with current departmental priorities and planned spending, is required to maintain these capabilities in the future. This includes, for example, fully funding Army prepositioned assets as planned and completing a planned reengineering of the C-5 aircraft.

In our previous reports concerning acquisition outcomes and best practices, we have noted the importance of matching warfighter requirements with available resources, a responsibility shared by the requirements and acquisition communities in DOD. As described in Air Force implementing guidance, there is within DOD a distinct separation between the requirements authority and acquisition authority. Under this guidance, this separation requires early and continued collaboration between both communities.

Analyses Used to Determine Mobility and Tanker Capabilities Were Inadequate

Analyses done for the MCS contained methodological limitations that create concerns about the adequacy and completeness of the study and decision makers approving the KC-X tanker proposal lacked required analyses identifying need and associated risk for a passenger and cargo capability.

Mobility Capabilities Study Limitations Raise Questions about Adequacy and Completeness

While DOD used an innovative approach in conducting the study and acknowledged some methodological limitations in its report, it did not fully disclose how these limitations could affect the MCS conclusions and recommendations. In September 2006, we reported that DOD’s conclusions were based, in some instances, on incomplete data and inadequate modeling and metrics that did not fully measure stress on the transportation system, and that, in some cases, MCS results were incomplete, unclear, or contingent on further study, making it difficult to identify findings and evaluate evidence. It is not clear how the analyses done for the study supported DOD’s conclusions, and we suggested that decision makers exercise caution in using the results of this study to make programmatic decisions.

As measured against relevant generally accepted research standards, we identified limitations in the MCS study and report that raise questions. Among our findings Aspects of modeling and data were inadequate in some areas because data were lacking and some of the models used could not simulate all relevant aspects of the missions. The report did not explain how these limitations could affect the study results or what the effect on the projected mobility capabilities might be. Relevant research standards require that models used are adequate for the intended purpose and represent a complete range of conditions, and also that data used are properly generated and complete. For example, the MCS modeled hypothetical homeland defense missions rather than missions for homeland defense demands from a well- defined and approved concept of operations for homeland defense because the specific details of the missions were still being determined, and DOD acknowledged that the data used may be incomplete. The MCS also was unable to model the flexible deterrent options/deployment order process to move units and equipment into theater due to lack of data, but the study assumed a robust use of this process, which in one scenario accounted for approximately 60 percent of the airlift prior to beginning combat operations. In addition, the MCS report contains more than 80 references to the need for improved modeling, and 12 of these references call for additional data or other refinements. Additionally, the MCS modeled the year 2012 to determine the transportation capabilities needed for the years 2007 through 2013. The year 2012 did not place as much demand for mobility assets in support of smaller military operations, such as peacekeeping, as other years. However, DOD officials considered 2012—the year modeled—as “most likely” to occur and stated that statistically it was not different from other years in the 2007 to 2013 period even though the number of smaller military operations is the least of any of the years reviewed.

As I mentioned, we have reported before on the lack of data available for analysis that could benefit decision makers. In September 2005, we reported that the Air Force captured data on short tons transported but did not systematically collect and analyze information on operational factors, such as weather and runway length, that impact how much can be loaded on individual missions. Therefore, Air Force officials could not know how often it met its secondary goal to use aircraft capacity as efficiently as possible. Without this information, Air Mobility Command officials do not know the extent to which opportunities exist to use aircraft more efficiently and whether operational tempo, cost, and wear and tear on aircraft could be reduced. In addition, DOD officials do not have the benefit of such analysis to determine future airlift requirements for planning purposes.

While the MCS concluded that combined U.S. and host nation transportation assets were adequate to meet U.S. objectives with acceptable risk, the report, in describing the use of warfighting metrics in its analyses, does not provide a clear understanding of the direct relationship of warfighting objectives to transportation capabilities. Acknowledging this point, the report stated that further analysis is required to understand the operational impact of increased or decreased strategic lift on achieving warfighting objectives. Relevant generally accepted research standards require that conclusions be supported by analyses. The use of warfighting metrics is a measure to determine whether combat tasks, such as achieving air superiority, are achieved. However, they do not measure whether appropriate personnel, supplies, and equipment arrived in accordance with timelines. As a result, we could not determine how the study concluded that planned transportation assets were adequate because the study did not contain a transparent analysis to support its conclusion or a clear roadmap in the report to help decision makers understand what that conclusion meant in terms of type and number of mobility assets needed. Previous DOD mobility studies primarily used mobility metrics, which measured success in terms of tons of equipment and personnel moved per day to accomplish military objectives. The use of both warfighting and mobility metrics to measure success would allow decision makers to know whether combat tasks were achieved and how much strategic transportation is needed to accomplish those tasks. In some cases, the MCS results were incomplete, unclear, or contingent on further study, making it difficult to identify findings and evaluate evidence. Relevant research standards require results to be presented in a complete, accurate, and relevant manner. For example, the report contains several recommendations for further studies and assessments, five of which are under way. However, at the time of our report, DOD had no plans to report the effect of these studies on the MCS results after the studies are complete. In addition, the report contains qualified information that is not presented clearly, such as varying assessments of intratheater assets in three different places in the report. The lack of clarity and conciseness of the reported results can limit the study’s usefulness to decision makers and stakeholders.

The MCS report also made recommendations to conduct further studies, develop plans and strategies, and improve data collection and mobility models. In fact, DOD officials told us at the time that a Mobility Capabilities Study-2006 was underway, as well as studies on intratheater lift, aerial refueling, and other mobility issues. However, unless DOD addresses the concerns I just outlined for you, decision makers may be unable to clearly understand the operational implications of the study results and make fully informed programmatic investment decisions concerning mobility capabilities. Also, some of the underlying assumptions used in the MCS have now changed significantly, such as the assumption that Army prepositioned equipment is in place and fully funded, which will no longer be the case. Therefore, the MCS analyses and results, which would be the starting point for any new studies, may no longer be relevant.

Mandatory Analyses Done to Support Passenger and Cargo Capabilities in the KC-X Tanker Were Incomplete

Mandatory Air Force policy requires Air Force organizations to use a formal capabilities-based approach to identify, evaluate, develop, field, and sustain capabilities that compete for limited resources. Contrary to mandatory Air Force implementing guidance, however, the Air Force proposal for a replacement refueling aircraft, the KC-X tanker, included a passenger and cargo capability without analyses identifying an associated gap, shortfall, or redundant capability. According to mandatory Air Force implementing guidance, analyses supporting the decision-making process should assess a capability based on the effects it seeks to generate and the associated operational risk of not having it. In this case, the supporting analyses determined neither need nor risk with regard to a passenger and cargo capability. Air Force officials could not provide supporting information sufficient to explain this discrepancy between the analyses and their proposal. Without sound analyses, the Air Force may be at risk of spending several billion dollars unnecessarily for a capability that may not be needed to meet a gap or shortfall.

Military decision makers approved the passenger and cargo capability as a requirement although supporting analyses identified no need or associated risk. Mandatory Air Force implementing guidance states that senior leaders must use the documented results of analyses to confirm the identified capability requirement. The Air Force Requirements for Operational Capabilities Council validated, and the Chairman of the Joint Chiefs of Staff’s Joint Requirements Oversight Council validated and approved, KCX tanker proposal with a passenger and cargo capability. Following the approvals of the oversight councils, DOD plans to solicit proposals and award a contract for the KC-X tanker late in fiscal year 2007. At this time, the Under Secretary of Defense for Acquisition, Technology and Logistics, who supervises DOD acquisition, must certify, as Milestone Decision Authority for the proposed tanker acquisition, that, among other things, the Joint Requirements Oversight Council has accomplished its statutory duties and that the proposed program is in compliance with DOD policies and regulations. However, the absence of analyses identifying a capability gap, shortfall, or redundancy, and the Joint Requirements Oversight Council approval of the program without these analyses is contrary to policy and implementing guidance and could preclude certification of the program by the Under Secretary. Absent this certification, the acquisition program for the KC-X tanker cannot begin.

In this report, we recommended that the Secretary of Defense direct the Secretary of the Air Force to accomplish the required analyses to evaluate the proposed passenger and cargo capability so as to determine if there is a gap, shortfall, or redundancy, assess the associated risk, and then submit such documentation to the Joint Requirements Oversight Council for validation. We also recommended that, once these analyses are completed, the Secretary of Defense direct the Chairman, Joint Chiefs of Staff, to formally notify the Under Secretary of Defense for Acquisition, Technology and Logistics that such analyses have been completed as required prior to certification of the program to Congress.

DOD disagreed with our first recommendation to accomplish the required analyses. In its comments, DOD stated that through the Joint Capabilities Integration and Development System process, the Air Force presented analysis and rationale for the passenger and cargo capability. DOD further stated that its Joint Requirements Oversight Council and the Air Force concluded that the analysis was sufficient justification for the capability and the Joint Requirements Oversight Council validated the requirement. However, as our report points out, DOD did not perform the required analyses and failed to identify a gap, shortfall, or redundancy for the passenger and cargo capability. Considering the requirement for analyses that separate needs from wants and the risk of unnecessary expenditures in this multi-year multi-billion dollar acquisition program, we continue to believe that our recommendation has merit and that the analyses required by mandatory guidance are necessary to inform the decision that begins the acquisition.

DOD agreed with our recommendation to formally notify the Under Secretary of Defense for Acquisition, Technology and Logistics once the required analyses have been completed. However, DOD did not offer assurance that the Air Force would accomplish the required analyses that evaluate the proposed passenger and cargo capability as we recommended, and then submit such documentation to the Joint Requirements Oversight Council for validation. We believe that the time it could take to accomplish the required analyses and submit the analyses for revalidation by the Joint Requirements Oversight Council, could delay the Under Secretary’s certification until just prior to the Milestone B decision, and may frustrate the congressional oversight that would otherwise be permitted under section 2366a. We believe that in a program committing $120 billion over several decades, the review confirming that needs are justified should occur as far in advance of program initiation as possible.

In light of the DOD comments on our report, we have put forward a matter for congressional consideration. Specifically, we are suggesting that Congress consider requiring in addition to the certification described by section 2366a of title 10, United States Code, the Under Secretary of Defense for Acquisitions, Technology and Logistics make a specific certification that the Air Force employed a sound, traceable, and repeatable process producing analyses that determined if there is a gap, shortfall, or redundancy and assessed the associated risk with regard to passenger and cargo capability for the KC- 135 Recapitalization, and consistent with service policy, these analyses are made available to the Joint Requirements Oversight Council prior to the Under Secretary’s certification of the program pursuant to section 2366a of title 10, United States Code. The Air Force intends to replace the fleet of more than 500 tankers and the Mobility Capabilities Study of 2005 set the requirement for tankers at a range of between 520 to 640 aircraft. Replacement of this fleet is estimated to cost a minimum of $72 billion. Compared to a refueling aircraft without a passenger and cargo capability, the inclusion of the capability is estimated, according to the Analysis of Alternatives done for the KC-X tanker, to increase costs by 6 percent. The Joint Requirements Oversight Council approval of the proposal of a replacement tanker aircraft with the passenger and cargo capability, without an established need supported by analyses and without an analysis of risk, could result in an unnecessary expenditure of at least $4.3 billion by our estimates.

In our August 1996 report, U.S. Combat Air Power: Aging Refueling Aircraft Are Costly to Maintain and Operate, we recommended consideration of a dual-use aircraft that could conduct both aerial refueling and airlift operations as a replacement for the KC-135. We recommended that the Secretary of Defense require that future studies and analyses of replacement airlift and tanker aircraft consider accomplishing the missions with a dual-use aircraft. DOD only partially concurred with this recommendation, expressing concern at that time about how a dual- use aircraft would be used and whether one mission area might be degraded to accomplish the second mission. The lack of analyses done to support the current proposal still does not give DOD officials information about how a dual-use aircraft would be used or whether the primary mission of aerial refueling would be degraded.

Employing Best Practices Can Shape Better Program Outcomes for DOD’s Airlift Acquisitions

Over the past 25 years, DOD has invested more than $140 billion on its airlift and tanker forces. Success for acquisitions requires sound decisions to ensure that program investments are getting promised returns—on time deliveries to the field, predictable costs, and sufficient capability. We have reviewed four major airlift programs and found they did not meet delivery schedules and were over cost. These programs did not involve huge technological leaps but presented significant design challenges to integrate new systems into the older aircraft. A consistent problem plaguing the programs was an insufficient job of analyzing the requirements and resources at the programs’ outset, a key systems engineering activity. The divergence between these programs’ experience and best product development practices are contributing factors to their outcomes.

Outcomes of Certain Airlift Programs

We assessed four airlift programs as part of our annual assessment of DOD’s major acquisition programs and each has experienced cost growth and schedule delays. Despite being based largely on low technological risks involving mature systems, these programs have failed to deliver on the business cases that justified their initial investment. DOD estimates it will need over $12 billion between 2007 and 2013 to develop, modify, or procure these aircraft. The specific airlift programs include The Air Force’s C-5 Avionics Modernization Program (AMP) is intended to improve the mission capability rate and transport capabilities, as well as reduce ownership costs by incorporating global air traffic management, navigation and safety equipment, modern digital equipment, and an all- weather flight control system.

The Air Force’s C-5 Reliability Enhancement and Reengining Program (RERP) is intended to enhance the reliability, maintainability, and availability of the C-5 through engine replacements and modifications to subsystems such as the electrical and fuel subsystems. The C-5 aircraft will require installation of the AMP capabilities before the aircraft engines can be replaced.

The Air Force’s C-130 Avionics Modernization Program (AMP) is intended to standardize the cockpit configurations and avionics of different models of C-130 aircraft by providing such things as communication and navigational system upgrades, terrain avoidance and warning system, dual flight management systems, and new data links.

The C-130J, the latest model of the C-130 aircraft series, is designed primarily for the transport of cargo and personnel within a theater of operation. Variants of the C-130J are being acquired by the Air Force (e.g., Air Mobility Command and Special Operations Command), Marine Corps, and Coast Guard.

Each of these programs has experienced problems that have impacted cost and schedule (see table 1). The net effect of the outcomes to date is that DOD is now paying more to modify or acquire these systems and the warfighter has had to wait longer than initially planned before new capability is delivered. For example, the Air Force now expects by 2011 to have completed the modification of about 135 fewer C-130 airlift aircraft when compared to its plan 2 years ago.

We anticipate there could be additional cost increases and schedule delays reported in the future. For example, the C-130 AMP fiscal year 2008 budget indicates that the total program costs have increased almost $700 million and planned quantities have been reduced from 434 units to 268 units— nearly doubling the program acquisition unit costs since December 2005. The program recently notified Congress of a critical Nunn-McCurdy breach concerning its unit cost increases. The budget also shows the Air Force plans to fund the modification of 110 C-5 aircraft with AMP improvements instead of 59 aircraft as stated in last year’s budget. According to C-5 RERP program officials, total program costs are expected to increase due to costs with the engine, pylons, and labor.

Airlift Programs Have Not Captured Critical Product Knowledge at Key Decision Points

Over the last several years, we have undertaken a body of work that examines weapon acquisition issues from the perspective that draws upon lessons learned from best commercial practices for product development. We have found that a key to successful product development is the formulation of a business case that provides demonstrated evidence that (1) the warfighter need exists and that it can best be met with the chosen concept and (2) the concept can be developed and produced within existing resources—including proven technologies, design knowledge, adequate funding, and adequate time to deliver the product when needed. The business case is then executed through an acquisition process that is anchored in knowledge. Leading firms ensure a high level of knowledge is achieved at key junctures in development, characterized as knowledge points described below: Knowledge point 1: A match must be made between the customer’s needs and the developer’s available resources—technology, engineering knowledge, time, and funding—before a program starts.

Knowledge point 2: The product’s design must be stable and must meet performance requirements before beginning system demonstration. This is primarily evidenced by the release of 90 percent of the design drawings by the critical design review and successful system integration.

Knowledge point 3: The product must be producible within cost, schedule, and quality targets and demonstrated to work as intended before production begins.

There is a synergy in this process, as the attainment of each successive knowledge point builds on the preceding one. We have found that if the knowledge based acquisition concept is not applied, a cascade of negative effects becomes magnified in the product development and production phases of an acquisition program leading to cost increases and schedule delays, poor product quality and reliability, and delays in getting new capability to the warfighter (see figure 2).

DOD programs often do not capture sufficient knowledge by critical junctures but decide to move forward regardless. The airlift systems we reviewed were not immune to this condition and have experienced unnecessary cost growth and schedule delays as a result. While we do not have in-depth knowledge on the specific details for these programs, we do have a broad understanding of the basic underpinnings that led to the problems. All of the programs were considered low technological risks by DOD because they planned to rely extensively on proven commercial and modified off the shelf technology for its new capabilities. However, these acquisitions have turned out to be more difficult than expected. The programs did not follow sound systems engineering practices for analyzing requirements and for ensuring a well integrated design at the right time. As a result, each program has encountered some difficulty in achieving design and production maturity as the program moved forward. Some of the causes to problems encountered include Failing to fully analyze the resources needed to integrate proven commercial technologies and subsystems into a military system before initiating development.

Not achieving a stable design before beginning system demonstration phase resulting in costly design changes and rework.

Failing to demonstrate the aircraft would work as required before making large production investments.

In all these instances where appropriate knowledge was not captured before moving forward, the impact has resulted in a predictable need for additional resources as shown below in specific airlift programs.

C-5 AMP

The C-5 AMP entered production without demonstrating that the system worked as intended and was reliable. The program entered production just 2 months after flight testing started and ran into significant design problems while trying to complete development. Problems uncovered after flight test began required modifications to the aircraft design which increased by 50 percent the number of engineering drawings needed for the system. Addressing these problems delayed the initial operational capability by a year and contributed to the significant growth in the program’s unit costs. Even today, 4 years after production was initiated, performance concerns remain with the C-5 AMP. The Director of Operational Test and Evaluation recently reported that the C-5 AMP is not operationally suitable because of high component failure rates, inadequate diagnostics systems, and low reliability rates.

C-5 RERP

The C-5 RERP did not demonstrate design stability before entering the system demonstration phase which resulted in rework and schedule delays. At the time the program entered system demonstration, program officials believed that they had released 90 percent of the design drawings but had not successfully demonstrated that the subsystems could be integrated onto the C-5 aircraft. During system integration activities the program found that the “pylon/thrust reverser” had to be redesigned to address overweight conditions and safety concerns. The program’s design efforts have also been hampered by the fact that its success is dependent upon the success of the C-5 AMP program. Presently, according to test officials, the C-5 AMP design is not mature enough to provide a baseline design for the RERP efforts. These design issues have contributed to an increase in costs and a 2-year delay in delivering an initial operational capability.

C-130 AMP

The C-130 AMP began development in 2001 without a clear understanding of the resources needed to integrate proven commercial technologies into a military system. According to the program office, they clearly underestimated the complexity of the engineering efforts that were needed to modify the different models of the C-130. At the critical design review held in 2005—the point that the design is expected to be stable and ready to begin the system demonstration phase—the program had not proven that the subsystems and components could be successfully integrated into the product.Upon integrating the new avionics into the test aircraft, program officials realized that it had significantly underestimated (by 400 percent) the amount of wiring and the number of harnesses and brackets needed for the installation. As a result, the design had to be reworked, delaying the delivery of the test aircraft and increasing costs.

C-130J

The Air Force procured the C-130J without assurances that the aircraft would work as intended. Program officials believed the design was mature when procurement began in 1996, largely because the C-130J evolved from earlier models and was offered as a commercial item. However, the C- 130J has encountered numerous deficiencies that had to be corrected in order to meet the minimum warfighter requirements delaying the initial aircraft delivery to the warfighter by about 1.5 years. DOD testing officials still report performance issues with the aircraft resulting in it being rated as partially mission capable. The performance issues involve the aircraft’s ability to meet its airdrop operations requirements, its effectiveness in non-permissive threat environments, and maintainability issues. Program officials plan to address the deficiencies as part of a C-130J modernization effort.

Conclusion

As we said at the beginning, our work shows that acquisition problems will likely persist until DOD provides a better foundation for buying the right things the right way. This involves making tough tradeoff decisions as to which programs should be pursued, and, more importantly, not pursued, making sure programs are executable, establishing and locking in needed requirements before programs are ever started, and making it clear who is responsible for what and holding people accountable when these responsibilities are not fulfilled. Recognizing this, DOD has tried to embrace best practices in its policies, as well as taking many other actions. However, DOD still has trouble distinguishing between wants and needs. Because of our concerns about the analyses done for both the MCS, which has broad implications for DOD’s mobility needs, and the KC-X tanker requirements, we would urge Congress and other decision makers to exercise caution when making airlift and tanker investment decisions.

DOD will continue to face challenges in modernizing its forces with new demands on the federal dollar created by changing world conditions. Consequently, it is incumbent upon DOD to find and adopt best product development practices that can allow it to manage its weapon system program in the most efficient and effective way. Success over the long term will depend on following knowledge-based acquisition practices as well as DOD leadership’s commitment to improving outcomes.

The four acquisition cases we cite in this testimony are not atypical for all programs. Even with no major technological invention necessary to meet the warfighters needs in these cases, acquisition outcomes are not good. There are consequences to these outcomes. The warfighter does not receive needed capability on time and the Department and Congress must spend additional unplanned money to correct mistakes—an expense they can ill afford. A knowledge-based product development process steeped in best practices from systems engineering can solve many of these problems before they start. DOD knows how to do this and, in fact, informs its acquisition policy with systems engineering rules. It should redouble its efforts to drive these policies into practice.

Mr. Chairman and members of the Subcommittee, this concludes our prepared statement. We would be pleased to answer any questions you may have.

Contact and Staff Acknowledgements

For further information about this statement, please contact William M. Solis at 202-512-8365 or Solisw@gao.gov or Michael J. Sullivan at 202-512- 4841 or Sullivanm@gao.gov. Contact points for Offices of Congressional Relations and Public Affairs may be found on the last page of this statement.

GAO staff who made major contributions to this testimony include Marie Ahearn, Ann Borseth, Cheryl Andrew, Claudia Dickey, Mike Hazard, Matthew Lea, Oscar Mardis, Sean Merrill, Karen Thornton, and Steve Woods.

Related GAO Products

Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368. Washington, D.C.: April 13, 2006 Defense Acquisitions: Assessments of Selected Major Weapon Programs. GAO-06-391. Washington, D.C.:March 31, 2006.

DOD Acquisition Outcomes: A Case for Change. GAO-06-257T. Washington, D.C.: November 15, 2005.

Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002.

Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001.

Defense Acquisition: Employing Best Practices Can Shape Better Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.

Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999.

Best Practices: Successful Application to Weapon Acquisition Requires Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.: February 24, 1998.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.