Fighter Aircraft: Better Cost Estimates Needed for Extending the Service Life of Selected F-16s and F/A-18s
Released: 2012-11-15T00:00:00

Letter

Background

The Air Force and the Navy plan to replace many of their current fighter aircraft with the Joint Strike Fighter, also called the F-35. DOD plans to buy a total of 2,443 F-35s—1,763 for the Air Force and 680 for the Department of the Navy—between 2008 and 2037. However, we have reported on F-35 issues since 2005 and found that the F-35 is still in development and the program has experienced significant delays and cost increases. For example, we reported that the F-35 program has experienced numerous delays in development and production, including program restructuring, which has increased time for development. Further, we reported that the F-35 program costs have increased 42 percent from the 2007 baseline and unit costs have doubled since the start of development in 2001. In addition, the DOD Fiscal Year 2011 Annual Report on Operational Test and Evaluation stated that the F-35 is not on track to meet operational effectiveness or operational suitability requirements and that testing identified structural and maintenance issues.first few production lots will need a significant number of modifications and upgrades in order to attain planned service life and capabilities. Due to delays in the F-35 program, the services have decided to extend the service life of some existing aircraft in order to maintain desired inventory levels.

Further, the DOD report also stated that aircraft produced in the The upgrade and service-life extension programs will likely be costly and may be subject to further changes. Our prior work has found that a high- quality cost estimate is critical to the success of any program because it can provide the basis for informed investment decision making, realistic budget formulation, and proactive course correction when warranted. The March 2009 GAO Cost Estimating and Assessment Guide is a compilation of best practices that federal cost-estimating organizations and industry use to develop and maintain reliable cost estimates throughout the life of a program. The GAO cost guide reported that a high-quality cost estimate has four characteristics including comprehensive when it accounts for all life-cycle costs, is sufficiently detailed to ensure that costs are neither omitted nor double-counted, and identifies ground rules and assumptions; well-documented when supporting documentation explains the process, sources, and methods used to create the estimate; accurate when it is based on the assessment of the costs most likely to be incurred and adjusted for inflation; and credible when the level of confidence has been identified through a risk and uncertainty analysis, and a sensitivity analysis has been conducted that identified the effects on the estimate of changing key assumptions. Also, a cost estimate is credible when it has been cross- checked with an independent estimate. The GAO cost guide describes eight types of independent cost estimate reviews which vary in the depth of analysis, ranging from a document review— merely assessing the estimate’s documentation—to an independently developed cost estimate—conducted by an organization outside the acquisition or program office. The Air Force and Navy each have service cost-estimating agencies, which develop independent cost estimates for major acquisition programs and other programs when requested.

Air Force and Navy Programs Would Extend the Service Life of 450 Aircraft at a Cost of Nearly $5 billion

The Air Force currently plans to upgrade and extend the service life of 300 F-16 aircraft at an estimated cost of $2.61 billion and the Navy plans to extend the service life of 150 F/A-18 aircraft at an estimated cost of $2.19 billion. Service officials have said that if the F-35 program experiences further delays, the services have the ability to expand the number of aircraft that are included in these programs.

Air Force Program Would Add 2,000 Hours to 300 F- 16s

In 2009, DOD and the Air Force were directed by statute and a committee report to provide reports on force-structure plans including alternatives such as buying aircraft and upgrading and extending the service life of selected aircraft. The Air Force reported on assessments of seven alternatives, which included buying between 150 and 300 new F-15Es, F/A-18 E/Fs, and F-16s, and upgrading and extending the service life of selected F-16s. The Air Force concluded that the cost for upgrading and extending the service life of current F-16s by 2,000 hours each would be 10 to 15 percent of the cost of procuring new F-16s, F-15Es, or F/A-18s. Extending the life of existing aircraft would provide 6 to 8 years of additional service and, according to the Air Force, provide essentially the same capability over that period as buying new legacy aircraft. In reviewing these reports, GAO concluded in 2011 that the analyses done were limited in part by the absence of F-16 durability and viability data.Figure 1 below shows the Air Force’s estimated costs for the current plans for capability upgrades and the service-life extension.

The Air Force estimates to upgrade capabilities and extend the service life of selected F-16 aircraft included research, development, test and evaluation, and procurement costs. The capability upgrades, Air Force officials explained, are needed to maintain mission effectiveness and will include, for example, an improved radar and data-link enhancements. The service-life extension will add 2,000 flight hours, or about 6 to 8 years, depending on flying conditions, to each aircraft. Most of the $2.61 billion to upgrade and extend the service life—$1.88 billion or 72 percent—is planned to be incurred in fiscal years 2018 through 2022. The remaining $722 million is programmed in the fiscal year 2013-2017 Future Years Defense Program.

Since the 2010 reports to Congress were completed, the Air Force has stated that without the service-life extension it would have to start gradually grounding some F-16s beginning in 2017. The Air Force is conducting tests to clarify the extent of the work required to extend the service life, and plans to begin work in fiscal year 2016. Also, the Air Force plans a milestone program review for the service-life extension in the first quarter of fiscal year 2013 and a milestone program review for the capability-upgrade program in the fourth quarter of fiscal year 2013. The Air Force plans to complete all the upgrade and service-life extension work by the early 2020s. Finally, Air Force officials said that they could expand these programs to include up to 650 aircraft if needed to attain desired inventory levels.

Navy Program Would Add 1,400 Hours to 150 F/A-18s

In response to a statutory requirement, in May 2011, the Navy submitted the results of a cost-benefit analysis comparing extending the service life of existing F/A-18 aircraft with procuring additional F/A-18 E/F aircraft. The Navy assessed six alternatives, which included various combinations of extending the service life of up to 280 F/A-18 A-D aircraft and procuring up to an additional 70 F/A-18 E/Fs. The Navy concluded that extending the service life of 150 F/A-18 A-D aircraft and buying 41 new F/A-18E/F aircraft would provide an acceptable inventory at a manageable level of risk.year 2018. Figure 2 below shows the Navy’s estimated cost for the service-life extension of selected F/A-18 aircraft.

The Navy plans to complete the service-life extension work in fiscal The Navy’s estimate includes procurement costs for materials and installation to extend the service life of 150 F/A-18s for an additional 1,400 hours each or about 5 years depending on flying conditions.

Almost all the Navy’s estimated $2.19 billion is programmed in the fiscal year 2013-2017 Future Years Defense Program. Finally, the Navy’s analysis showed that the 150 aircraft selected for the service-life extension will individually be evaluated for capability upgrades—such as adding the ability to integrate with newer aircraft—as well as upgrades of other critical components reaching the end of their service life and becoming obsolete. After evaluating the aircraft, the Navy will decide whether to do this additional work. Finally, Navy officials said that they could expand their program if needed, to include up to 280 aircraft. Navy officials explained that they will review the service-life extension effort annually as part of an overall review of the F/A-18 program.

Air Force and Navy Cost Estimates Exhibit Many Characteristics of a High-Quality Cost Estimate but Do Not Reflect Some Significant Potential Costs

The Air Force’s and Navy’s cost estimates to upgrade and extend the service life of selected F-16 and F/A-18 aircraft exhibit many cost- estimating characteristics and best practices of a high-quality cost estimate, but do not reflect all the potential total costs that may occur. We assessed the Air Force estimate for capability upgrades and the Air Force estimate for the service-life extension of selected F-16 aircraft, which were prepared for the fiscal year 2013 budget request; and the Navy estimate for the service-life extension of selected F/A-18 aircraft. Overall, we found that the Air Force and Navy followed many of the best practices that support the four characteristics of a high-quality estimate. However, both the Air Force and Navy estimates were not fully credible due to two shortcomings—the estimates did not show the range of potential costs that are likely to be incurred and were not validated by comparison with an independently developed estimate. As a result, decision-makers do not have visibility of how much the total costs will be and how they may increase if program quantities increase or additional work is required on some aircraft, which could hinder their ability to assess budgets and affordability.

Air Force Estimates Exhibit Some Characteristics of a High- Quality Estimate but Are Not Fully Credible

The Air Force’s $1.79 billion estimate for capability upgrades and the $820 million estimate for service-life extension were well-documented, accurate, and mostly comprehensive. However, the estimates were not fully credible because the Air Force’s analysis did not clearly show the potential range of total costs that would occur if more aircraft are included in the programs and the estimates were not compared to an independently developed estimate. In assessing the extent to which the Air Force’s cost estimates exhibited the four characteristics of a high- quality estimate, we found the following:

Comprehensive: The Air Force’s estimates were substantially comprehensive because they included all the work planned for 300 aircraft, identified key cost-estimating ground rules and assumptions, and included development and procurement costs.

Well-documented: Both estimates were well-documented because they identified the source of the data, included the work that is planned, documented the cost-estimating methodologies, showed how the calculations were made, and were reviewed by management.

Accurate: Both estimates were accurate because they were adjusted for inflation, based on an assessment of most likely costs for 300 aircraft, and used data from comparable programs where appropriate.

Credible: For both estimates, the Air Force cross-checked major cost elements to determine whether results were similar and conducted a risk and uncertainty analysis that included a list of the risks assessed. For example, the analysis assessed the risk that more work may be required to extend the service life than currently planned. Also, the Air Force identified some cost drivers, which are the items that have the greatest effect on the estimated cost, such as labor rates. While the Air Force estimates exhibited many characteristics of a high-quality estimate, they had two shortcomings that lessened the estimates’ credibility. First, the analysis did not include an estimate of the total costs that may occur if more than 300 aircraft are included in the programs up to the potential maximum of 650 aircraft. Second, the estimates were not compared to an independently developed estimate.

Regarding the first shortcoming, best practices state that a credible cost estimate should include an assessment of how the cost estimate may change in response to changes in key program assumptions. This is known as a sensitivity analysis. Such an analysis helps decision makers identify areas that could significantly affect a program’s cost, such as changes in program quantities. According to Air Force officials, the service may choose to upgrade and extend the service life of additional aircraft if there are further delays in F-35 production. For example, in a written statement, Air Force leaders testified in March 2012 that the intent is to include 350 F-16 aircraft in the upgrade and life extension programs—50 more aircraft than originally planned. Air Force officials explained that they did not assess the range of costs that may result from expanding the number of aircraft in the program up to the maximum of 650 since the program is currently approved for only 300 aircraft. However, without an assessment of how much the total cost may increase as the aircraft quantity increases, the estimate lacks transparency on the full range of possible costs, which could be significant. For example, we calculated that the cost to upgrade and extend the service life of all 650 aircraft could total $5.53 billion.

The second major shortcoming is that the Air Force did not compare the estimates to independently developed cost estimates. Having an independent estimate is considered one of the best and most reliable estimate validation methods since an independent estimate is an objective assessment of whether the program office estimate is reasonable and can be achieved. Further, our past work has shown that an independent estimate tends to be higher and more accurate than a program office estimate. Air Force officials explained that it is likely the Air Force Cost Analysis Agency will be requested to develop a formal independent cost estimate for upcoming program reviews, but the decision to require an independent estimate has not yet been made. Further, the program review for the capability-upgrade program will not be held for a year—it is currently scheduled for the fourth quarter of fiscal year 2013. Until the Air Force obtains an independent cost estimate, decision makers will not have the assurance that the programs’ estimated cost can be relied upon for making budgeting and trade-off decisions. Furthermore, a program that has not been reconciled with an independent estimate has an increased risk of proceeding underfunded because an independent estimate provides an objective and unbiased assessment of whether the program estimate can be achieved.

Navy Estimate Exhibits Characteristics of a High- Quality Estimate but Does Not Capture Costs of Additional, Nonstructural Work

The Navy’s $2.19 billion cost estimate for the service-life extension of F/A-18s was comprehensive, well-documented, and accurate. However, the estimate was not fully credible because it did not include an assessment of the costs for additional work on the aircraft that may be done at the same time as the service-life extension and the estimate was not compared to an independently developed estimate. Since the Navy will assess on an aircraft-by-aircraft basis whether to do this additional work, the costs would be in addition to the $2.19 billion for the service-life extension and most of these costs are not included in the Navy’s spending plans through fiscal year 2017. In assessing the extent to which the Navy’s cost estimate exhibited the four characteristics of a high- quality estimate, we found the following:

Comprehensive: The Navy’s estimate identified key cost-estimating ground rules and assumptions. Also, the Navy’s estimate included the costs to buy and install the materials needed to repair the airframe so that it can fly the additional 1,400 hours. Finally, the Navy’s analysis included assessing the costs for different quantities from 150 up to 280 aircraft.

Well-documented: The estimate was well-documented since it described the methodology used and the calculations performed to develop the estimate so that a cost analyst unfamiliar with the program could replicate it. Also, the Navy’s documentation included steps to ensure data reliability, and the estimate was reviewed by management.

Accurate: The Navy’s estimate was accurate since it was adjusted for inflation, not overly conservative or optimistic, and incorporated data from comparable programs where appropriate.

Credible: Finally, the Navy’s estimate was partially credible because the Navy followed some, but not all, of the best practices for this characteristic. For example, the Navy conducted a sensitivity analysis which identified cost drivers, such as materiel and labor costs. Also, the Navy conducted a risk and uncertainty analysis which accounted for some risks, such as the risk that the service-life extension may require more work than currently planned. However, the Navy did not assess the potential range of costs for all work that might be done and focused only on the costs associated with extending the life of the airframe, which officials described as consistent with Navy guidance.

Specifically, the Navy told us it will assess on an aircraft-by-aircraft basis whether the F/A-18s also need capability upgrades to maintain mission effectiveness, such as adding the ability to integrate with newer aircraft.

According to the Navy, such capability upgrades could cost an average of about $1.76 million per aircraft, but uncertainty exists about how many aircraft the Navy will actually decide to upgrade and therefore the total cost associated with these upgrades is uncertain. In addition, the Navy has determined that some of the F/A-18 aircraft may also require life extension or replacement of nonstructural parts or systems that are becoming obsolete. The Navy estimated that this work could average $5.64 million per aircraft. Navy officials explained that they will decide on an aircraft-by-aircraft basis whether to do this work when the aircraft is inspected just prior to beginning service-life extension. Navy officials told us that if an aircraft needs a lot of this type of work, the Navy may decide not to extend the service life of that particular aircraft, but rather substitute another one in its place. Given the uncertainties, the costs associated with this additional work are not included in the Navy’s $2.19 billion estimate nor are they included in the Navy’s spending plans through fiscal year 2017. The purpose of a sensitivity analysis is to identify the effects on the estimate of changing key factors and to show a resulting range of possible costs. Further analysis could show the likelihood of the Navy incurring some or all of the additional costs. Without an assessment of the effect on the total costs that would be incurred if some of the 150 aircraft need additional work at an additional cost, decision makers will not know the full range of potential costs.

Finally, the Navy did not compare the estimate to an independently developed cost estimate such as one that could have been requested from the Naval Center for Cost Analysis. Navy officials stated that the estimate was reviewed extensively within the Navy and that an independent estimate was not required because the service-life extension is a modification of an existing aircraft and is not a separate acquisition program. Even if not required, Navy officials said that they could have requested the Naval Center for Cost Analysis to develop an independent estimate for this program. In our prior work, we have found that an independent cost estimate is considered to be one of the most reliable validation methods. If the Navy’s estimate were validated by an independent cost estimate, decision-makers could place more credibility in the estimate.

Conclusions

The Air Force and the Navy decided to upgrade and extend the service life of selected F-16 and F/A-18 aircraft to provide a capability and capacity “bridge” until the F-35 is available in sufficient numbers. The cost to do so is already estimated to total almost $5 billion and likely will increase due to slippage in the F-35 program and increases in program scope. In many respects, the Air Force and Navy cost estimates we evaluated were developed in accordance with some best practices. Specifically, we found the estimates to be well-documented and accurate and mostly comprehensive. However, the Air Force and Navy did not follow some important best practices and as a result did not clearly identify the potential total costs that may result as the programs evolve. For example, without an analysis that includes the potential for increases in the number of aircraft in the program or that clearly identifies the cost for additional work that may be required on some of the aircraft, decision makers are not aware of how much total costs could increase if these events occur. Further, unless the services reconcile the program cost estimate with an independent cost estimate, there is an increased risk that the actual costs will exceed the estimate. According to the GAO Cost Estimating and Assessment Guide, independent estimates are usually higher and more accurate than program estimates and therefore if a program estimate is close to an independently developed estimate, one can be more confident that the estimate is credible. The guide further explains that there are various types of independent cost estimate reviews that can be performed, including seeking an independent estimate. Since these are multibillion-dollar investments that are critical to maintaining fighter capacity, an independent review of service estimates or an independently-developed estimate—which could be performed by the respective service cost-estimating agencies—would provide assurance about the likely full costs. Without fully credible cost estimates, service and congressional decision-makers will not have reasonable confidence of knowing the potential range of costs to upgrade and extend the service life of selected F-16 and F/A-18 aircraft as they make resource and trade-off decisions, develop and review budget requests, and assess the programs’ affordability.

Recommendations for Executive Action

To improve future updates of Air Force and Navy cost estimates for upgrading and extending the service life of selected F-16s and F/A-18s and to improve the ability of decision-makers to assess the potential total costs, we recommend that the Secretary of Defense take the following four actions: Direct the Secretary of the Air Force to update its cost estimates, and in doing so: include in its sensitivity analysis an assessment of the range of possible costs if the capability-upgrade and service-life extension programs are expanded to more than 300 aircraft including up to the maximum of 650 aircraft, and obtain an independent review of the updated cost estimates.

Direct the Secretary of the Navy to update its cost estimates and in doing so: include in its sensitivity analysis an assessment of the range of possible costs for extending the service life of other nonstructural components that are becoming obsolete and capability upgrades that may be required for some of the 150 aircraft, and obtain an independent review of the updated cost estimates.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD for comment. In written comments on the draft of this report, DOD agreed with all four recommendations. DOD’s comments are reprinted in their entirety in appendix III. DOD also provided technical comments, which we incorporated into the report as appropriate.

Regarding the first recommendation for improving future updates of the Air Force cost estimates, DOD concurred, stating that the Air Force had updated planned force structure requirements to upgrade and extend the service life of 50 aircraft in addition to the original 300 aircraft and therefore estimated costs for 350 aircraft. Also, the department stated that while these programs could be expanded beyond 350 aircraft, further cost excursions are premature until F-35 production has matured. The department stated that it would update its cost estimates with another sensitivity analysis containing an assessment of the range of possible costs if force structure requirements change further. While we acknowledge that estimating costs for 350 aircraft is a step in the right direction, an assessment of how much the total cost may increase up to the maximum of 650 aircraft is needed to provide transparency on the full range of costs, which could be significant—potentially totaling up to $5.53 billion. DOD also noted that further cost excursions are premature until F- 16 structural testing is complete. However, Air Force officials explained during our review that data from testing are available throughout the testing period and that they could use this information to update cost estimates before testing is completed in 2016. Until the Air Force completes analyses to show the potential range of costs, decision makers will continue to lack crucial data on how much total costs could increase as they make resource and trade-off decisions.

Regarding the second recommendation for improving future updates of the Air Force cost estimates by obtaining an independent review of the updated cost estimates, DOD agreed, stating that the Air Force Cost Analysis Agency had reviewed the estimates for the Air Force’s fiscal year 2014 budget development process and that, if further cost assessments are warranted, the Air Force would submit the estimates for an independent review. An independent review of cost estimates is important because a program that has not been reconciled with an independent estimate has an increased risk of proceeding underfunded since an independent estimate provides an objective and unbiased assessment of whether the program estimate can be achieved. As the Air Force updates its cost estimates for these $2.61 billion programs, up- to-date, independently validated cost estimates would provide decision- makers assurance that the programs’ estimated cost can be relied upon as they assess program affordability.

Regarding the two recommendations for improving future updates of the Navy cost estimates, DOD concurred but did not explain when or what specific actions it intends to take to implement these recommendations. We are encouraged that the department agrees these are important, valid steps to take and further believe that it is important for the Navy to implement these recommendations in a timely manner to facilitate review and implementation of these programs since the programs are estimated to cost $2.19 billion and could increase an average of $5.64 million per aircraft.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretary of the Air Force, and the Secretary of the Navy. The report also is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (404) 679-1816 or pendletonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix IV.

Appendix I: Scope and Methodology

To describe the Air Force’s and the Navy’s plans to upgrade and extend the service life of selected F-16s and F/A-18s, we reviewed the services’ documentation of the programs’ purpose and scope including the underlying analysis of alternative approaches. To identify the Air Force’s and the Navy’s estimated costs to upgrade and extend the service life of F-16s and F/A-18s, we obtained the services’ cost estimates for these programs and obtained any updates to these estimates. The Air Force updated its initial estimate to support development of the fiscal year 2013 budget. The Navy’s cost estimate was reported to Congress in May 2011. All of the cost estimates from these sources were converted to fiscal year 2013 dollars using the indexes published by the Office of the Secretary of Defense (Comptroller) in the National Defense Budget Estimates for Fiscal Year 2013, commonly referred to as the “Green Book”.

To assess the extent to which the cost estimates exhibited characteristics of a high-quality cost estimate, we compared how the estimates were developed to the cost estimating best practices outlined in the GAO Cost Estimating and Assessment Guide. The guide is a compilation of best practices that federal cost-estimating organizations and industry use to develop and maintain reliable cost estimates that management can use for making informed decisions. The guide describes 19 best practices for developing a comprehensive, well-documented, accurate, and credible cost estimate—the four characteristics of a high-quality cost estimate. We analyzed the extent to which the Air Force and Navy cost estimates met each of these four characteristics and assigned each a rating of not met, The overall rating minimally met, partially met, substantially met, or met.for each characteristic was based on the average rating of the best practices for each characteristic. We also held detailed discussions with service officials and reviewed program documentation to identify key factors that could affect the potential total costs such as changes in aircraft quantities or additional work to maintain aircraft mission effectiveness that may not have been included in the estimates. We shared with program officials our cost guide, the criteria against which we evaluated the program’s estimated cost, as well as our preliminary findings. When warranted, we updated our analyses on the basis of the agency response and additional documentation that was provided to us. Finally, we corroborated our analyses in interviews with service officials responsible for developing the cost estimates.

We conducted our work at the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics; Office of the Chief of Staff of the Air Force, Directorate of Force Application; Ogden Air Logistics Center at Hill Air Force Base; F-16 System Program Office at Wright- Patterson Air Force Base; Air Combat Command; Air Force Cost Analysis Agency; Air Force Fleet Viability Board; Office of the Deputy Assistant Secretary of the Navy for Air Programs; Headquarters United States Marine Corps; Naval Air Systems Command; and the Naval Center for Cost Analysis.

We conducted this performance audit from May 2011 to November 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings, and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Air Force Cost Estimates for Upgrading and Extending the Service Life of Selected F-16 Aircraft in Then-Year Dollars

Dollars in millions

The Navy estimate was reported as a cost per aircraft in fiscal year 2011 dollars. Therefore, there is not comparable data for showing the Navy estimate in then-year dollars.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, the following staff members made key contributions to this report: Patricia W. Lentini, Assistant Director; Brenda M. Waterfield; Benjamin D. Thompson; Joseph M. Capuano; Ophelia Robinson; Karen A. Richey; Jennifer K. Echard; Charles W. Perdue; Michael D. Silver; Michael Shaughnessey; and Richard S. Powelson.