Fighter Aircraft: Better Cost Estimates Needed for Extending the Service Life of Selected F-16s and F/A-18s
Published: 2012-11-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Fighter aircraft are important to achieve and maintain air dominance during combat operations as well as to protect the homeland. DOD plans to replace many of its current fighter fleet with the F-35; however, the F-35 program has experienced numerous delays and cost increases. To maintain fighter capabilities and capacity, the Air Force and Navy have decided to upgrade and extend the service life of selected F-16 and F/A-18 aircraft. In this context, two subcommittees of the House Armed Services Committee asked GAO to (1) describe the Air Force and Navy plans to upgrade and extend the service life of selected F-16 and F/A-18 aircraft; and (2) assess the extent to which cost estimates for these upgrades and life extensions exhibit characteristics of a high-quality cost estimate. GAO obtained documentation of the plans and estimates, compared the estimates to best practices outlined in the GAO Cost Estimating Guide, and assessed factors that could affect total costs.
section_title: What GAO Found
paragraphs: The Air Force plans to upgrade and extend the service life of 300 F-16 aircraft and the Navy 150 F/A-18 aircraft, at a combined cost estimated at almost $5 billion in fiscal year 2013 dollars.
The Air Force plans to extend the service life of selected F-16s by 2,000 flying hours each as well as install capability upgrades such as an improved radar. The Air Force estimates that it will complete this work by 2022 at a cost of $2.61 billion. About 28 percent of the projected costs are included in the Air Force's spending plans through 2017, with the remainder expected to be incurred in 2018-2022.
The Navy plans to extend the service life of selected F/A-18s by 1,400 flying hours each and may install capability upgrades on some of the 150 aircraft--such as adding the ability to integrate with newer aircraft. The Navy projects that it will complete the life extension by 2018 at a cost of $2.19 billion, with most of these costs included in its spending plans through 2017, but costs associated with any upgrades are not included in the Navy estimate or in its spending plans.
Air Force and Navy officials told GAO that they could ultimately extend the service life of up to 650 F-16s and 280 F/A-18s if needed to attain desired inventory levels.
The Air Force's and Navy's cost estimates to upgrade and extend the service life of selected fighter aircraft exhibit some characteristics of a high-quality cost estimate but do not reflect all potential costs. The estimates were: well-documented since they identified data sources and methodologies; accurate since they accounted for inflation and were checked for errors; and mostly comprehensive since they included the work planned and identified key assumptions. However, the estimates were not fully credible in part because they did not assess the extent to which the total costs could change if additional work is done or more aircraft are included in the programs. Another factor affecting the credibility of the estimates is that they have not been compared to an independently developed estimate. GAO's past work has shown that such an independent cost estimate is one of the best validation methods since an independent cost estimate tends to be higher and more accurate than a program office estimate. Air Force and Navy officials told GAO that they use Department of Defense and military department guidance that allows for some variation in how the estimates are developed depending on the dollar value and maturity of the program. However, these programs--which are critical to maintain fighter capability and capacity as current inventory ages--total almost $5 billion and the costs will increase if program quantities and scope increase. Without fully credible cost estimates, including an analysis of how much total costs may increase, decision makers will not have visibility into the range of potential costs, which could hinder their ability to formulate realistic budgets and make informed investment decisions.
section_title: What GAO Recommends
paragraphs: GAO recommends that the Air Force and Navy follow all best practices to enhance the credibility of the cost estimates for the F-16 and F/A-18 upgrades and life extensions including an assessment of the potential range of costs and seeking independent cost estimates.

section_title: Letter
paragraphs: Fighter aircraft are important to achieving and maintaining air dominance  during combat operations as well as protecting the homeland. These  aircraft operate during the first days of a conflict to penetrate enemy air  space and defeat air defenses, which allows follow-on forces freedom to  operate. Overall, the Air Force and Navy have about 3,500 fighter aircraft  in inventory as of fiscal year 2012, including about 1,020 Air Force F-16s   Many of the Department of Defense’s  and about 624 Navy F/A-18 A-Ds.(DOD) current fighter aircraft are more than 20 years old on average, and  although DOD plans to replace much of the existing inventory by  procuring 2,443 new F-35s, the department has experienced numerous  delays and cost increases in the F-35 program. Therefore, the Air Force  and the Navy have considered several alternatives to maintain desired  fighter inventory levels at an acceptable level of risk and to maintain  fighter capabilities and capacity as a “bridge” until replacement F-35  aircraft enter service in sufficient numbers. The Air Force has decided to  upgrade and extend the service life of selected F-16s and the Navy has  decided to extend the service life of selected F/A-18s. These programs  are estimated to cost several billion dollars at a time when DOD is  simultaneously facing the competing demands of developing and  procuring F-35s at a cost approaching $400 billion, supporting ongoing  operations, and implementing $487 billion of reductions over the next 10  years, which the department has estimated that it needs to cut in order to  comply with the Budget Control Act of 2011.
We previously have reported that the F-35 program has experienced  numerous delays in development and production, including program  restructuring, which has added time for development. As a result, DOD  has deferred procurement of 410 F-35s until after fiscal year 2017.  Considering that further slips in the F-35 program may occur and could  increase the scope and cost of the services’ plans for upgrading and  extending the service life of current aircraft, you asked us to (1) describe  the Air Force and Navy plans to upgrade and extend the service life of  selected F-16 and F/A-18 aircraft, and (2) assess the extent to which cost  estimates for these programs exhibit characteristics of a high-quality cost  estimate. This report reviewed estimates of costs projected for fiscal  years 2011 through 2022.
To describe the Air Force’s and the Navy’s plans to upgrade and extend  the service life of selected F-16s and F/A-18s, we reviewed the services’  documentation of the programs’ purpose and scope including the  underlying analysis of alternative approaches. We also obtained cost  estimates for these programs and obtained any updates to these  estimates. Since the service cost estimates were developed in different  base years, all of the cost estimates in this report have been converted to  fiscal year 2013 dollars using the indexes published by the Office of the  Under Secretary of Defense (Comptroller) so that the data can be  presented in comparable terms. To assess the extent to which cost  estimates exhibit characteristics of a high-quality cost estimate, we  compared how the estimates were developed to the cost-estimating best  practices outlined in the GAO Cost Estimating and Assessment Guide.  We analyzed the extent to which the Air Force and Navy cost estimates  were comprehensive, well-documented, accurate, and credible—the four  characteristics of a high-quality cost estimate—and assigned each  characteristic a rating of not met, minimally met, partially met,  substantially met, or met. We also held detailed discussions with service  officials and reviewed program documentation to identify key factors that  could affect the potential total costs such as changes in aircraft quantities  or additional work to maintain aircraft mission effectiveness that may not  have been included in the estimates. We shared our cost guide, the  criteria against which we evaluated the program’s cost estimates, and our  preliminary findings with program officials. When warranted, we updated  our analyses based on the agency response and additional  documentation provided to us. Finally, we corroborated our analyses in  interviews with service officials responsible for developing the cost  estimates. See appendix I for a complete description of our scope and  methodology.
We conducted this performance audit from May 2011 to November 2012  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The Air Force and the Navy plan to replace many of their current fighter  aircraft with the Joint Strike Fighter, also called the F-35. DOD plans to  buy a total of 2,443 F-35s—1,763 for the Air Force and 680 for the  Department of the Navy—between 2008 and 2037. However, we have  reported on F-35 issues since 2005 and found that the F-35 is still in  development and the program has experienced significant delays and  cost increases. For example, we reported that the F-35 program has  experienced numerous delays in development and production, including  program restructuring, which has increased time for development.  Further, we reported that the F-35 program costs have increased 42  percent from the 2007 baseline and unit costs have doubled since the  start of development in 2001. In addition, the DOD Fiscal Year 2011  Annual Report on Operational Test and Evaluation stated that the F-35 is  not on track to meet operational effectiveness or operational suitability  requirements and that testing identified structural and maintenance  issues.first few production lots will need a significant number of modifications  and upgrades in order to attain planned service life and capabilities. Due  to delays in the F-35 program, the services have decided to extend the  service life of some existing aircraft in order to maintain desired inventory  levels.
Further, the DOD report also stated that aircraft produced in the  The upgrade and service-life extension programs will likely be costly and  may be subject to further changes. Our prior work has found that a high- quality cost estimate is critical to the success of any program because it  can provide the basis for informed investment decision making, realistic  budget formulation, and proactive course correction when warranted. The  March 2009 GAO Cost Estimating and Assessment Guide is a  compilation of best practices that federal cost-estimating organizations  and industry use to develop and maintain reliable cost estimates  throughout the life of a program. The GAO cost guide reported that a  high-quality cost estimate has four characteristics including  comprehensive when it accounts for all life-cycle costs, is sufficiently  detailed to ensure that costs are neither omitted nor double-counted,  and identifies ground rules and assumptions; well-documented when supporting documentation explains the  process, sources, and methods used to create the estimate; accurate when it is based on the assessment of the costs most likely  to be incurred and adjusted for inflation; and  credible when the level of confidence has been identified through a  risk and uncertainty analysis, and a sensitivity analysis has been  conducted that identified the effects on the estimate of changing key  assumptions. Also, a cost estimate is credible when it has been cross- checked with an independent estimate. The GAO cost guide  describes eight types of independent cost estimate reviews which  vary in the depth of analysis, ranging from a document review— merely assessing the estimate’s documentation—to an independently  developed cost estimate—conducted by an organization outside the  acquisition or program office. The Air Force and Navy each have  service cost-estimating agencies, which develop independent cost  estimates for major acquisition programs and other programs when  requested.
section_title: Air Force and Navy Programs Would Extend the Service Life of 450 Aircraft at a Cost of Nearly $5 billion
paragraphs: The Air Force currently plans to upgrade and extend the service life of  300 F-16 aircraft at an estimated cost of $2.61 billion and the Navy plans  to extend the service life of 150 F/A-18 aircraft at an estimated cost of  $2.19 billion. Service officials have said that if the F-35 program  experiences further delays, the services have the ability to expand the  number of aircraft that are included in these programs.
subsections: section_title: Air Force Program Would Add 2,000 Hours to 300 F- 16s
paragraphs: In 2009, DOD and the Air Force were directed by statute and a committee  report to provide reports on force-structure plans including alternatives  such as buying aircraft and upgrading and extending the service life of  selected aircraft. The Air Force reported on assessments of seven  alternatives, which included buying between 150 and 300 new F-15Es,  F/A-18 E/Fs, and F-16s, and upgrading and extending the service life of  selected F-16s. The Air Force concluded that the cost for upgrading and  extending the service life of current F-16s by 2,000 hours each would be  10 to 15 percent of the cost of procuring new F-16s, F-15Es, or F/A-18s.  Extending the life of existing aircraft would provide 6 to 8 years of  additional service and, according to the Air Force, provide essentially the  same capability over that period as buying new legacy aircraft. In  reviewing these reports, GAO concluded in 2011 that the analyses done were limited in part by the absence of F-16 durability and viability data.Figure 1 below shows the Air Force’s estimated costs for the current  plans for capability upgrades and the service-life extension.
The Air Force estimates to upgrade capabilities and extend the service  life of selected F-16 aircraft included research, development, test and  evaluation, and procurement costs. The capability upgrades, Air Force  officials explained, are needed to maintain mission effectiveness and will  include, for example, an improved radar and data-link enhancements.  The service-life extension will add 2,000 flight hours, or about 6 to 8  years, depending on flying conditions, to each aircraft. Most of the $2.61  billion to upgrade and extend the service life—$1.88 billion or 72  percent—is planned to be incurred in fiscal years 2018 through 2022. The  remaining $722 million is programmed in the fiscal year 2013-2017 Future  Years Defense Program.
Since the 2010 reports to Congress were completed, the Air Force has  stated that without the service-life extension it would have to start  gradually grounding some F-16s beginning in 2017. The Air Force is  conducting tests to clarify the extent of the work required to extend the  service life, and plans to begin work in fiscal year 2016. Also, the Air  Force plans a milestone program review for the service-life extension in  the first quarter of fiscal year 2013 and a milestone program review for  the capability-upgrade program in the fourth quarter of fiscal year 2013.  The Air Force plans to complete all the upgrade and service-life extension  work by the early 2020s. Finally, Air Force officials said that they could  expand these programs to include up to 650 aircraft if needed to attain  desired inventory levels.
section_title: Navy Program Would Add 1,400 Hours to 150 F/A-18s
paragraphs: In response to a statutory requirement, in May 2011, the Navy submitted  the results of a cost-benefit analysis comparing extending the service life  of existing F/A-18 aircraft with procuring additional F/A-18 E/F aircraft.  The Navy assessed six alternatives, which included various combinations  of extending the service life of up to 280 F/A-18 A-D aircraft and procuring  up to an additional 70 F/A-18 E/Fs. The Navy concluded that extending  the service life of 150 F/A-18 A-D aircraft and buying 41 new F/A-18E/F  aircraft would provide an acceptable inventory at a manageable level of  risk.year 2018. Figure 2 below shows the Navy’s estimated cost for the  service-life extension of selected F/A-18 aircraft.
The Navy plans to complete the service-life extension work in fiscal  The Navy’s estimate includes procurement costs for materials and  installation to extend the service life of 150 F/A-18s for an additional  1,400 hours each or about 5 years depending on flying conditions.
Almost all the Navy’s estimated $2.19 billion is programmed in the fiscal  year 2013-2017 Future Years Defense Program. Finally, the Navy’s  analysis showed that the 150 aircraft selected for the service-life  extension will individually be evaluated for capability upgrades—such as  adding the ability to integrate with newer aircraft—as well as upgrades of  other critical components reaching the end of their service life and  becoming obsolete. After evaluating the aircraft, the Navy will decide  whether to do this additional work. Finally, Navy officials said that they  could expand their program if needed, to include up to 280 aircraft. Navy  officials explained that they will review the service-life extension effort  annually as part of an overall review of the F/A-18 program.
section_title: Air Force and Navy Cost Estimates Exhibit Many Characteristics of a High-Quality Cost Estimate but Do Not Reflect Some Significant Potential Costs
paragraphs: The Air Force’s and Navy’s cost estimates to upgrade and extend the  service life of selected F-16 and F/A-18 aircraft exhibit many cost- estimating characteristics and best practices of a high-quality cost  estimate, but do not reflect all the potential total costs that may occur. We  assessed the Air Force estimate for capability upgrades and the Air Force  estimate for the service-life extension of selected F-16 aircraft, which  were prepared for the fiscal year 2013 budget request; and the Navy  estimate for the service-life extension of selected F/A-18 aircraft. Overall,  we found that the Air Force and Navy followed many of the best practices  that support the four characteristics of a high-quality estimate. However,  both the Air Force and Navy estimates were not fully credible due to two  shortcomings—the estimates did not show the range of potential costs  that are likely to be incurred and were not validated by comparison with  an independently developed estimate. As a result, decision-makers do  not have visibility of how much the total costs will be and how they may  increase if program quantities increase or additional work is required on  some aircraft, which could hinder their ability to assess budgets and  affordability.
subsections: section_title: Air Force Estimates Exhibit Some Characteristics of a High- Quality Estimate but Are Not Fully Credible
paragraphs: The Air Force’s $1.79 billion estimate for capability upgrades and the  $820 million estimate for service-life extension were well-documented,  accurate, and mostly comprehensive. However, the estimates were not  fully credible because the Air Force’s analysis did not clearly show the  potential range of total costs that would occur if more aircraft are included  in the programs and the estimates were not compared to an  independently developed estimate. In assessing the extent to which the  Air Force’s cost estimates exhibited the four characteristics of a high- quality estimate, we found the following:
Comprehensive: The Air Force’s estimates were substantially  comprehensive because they included all the work planned for 300  aircraft, identified key cost-estimating ground rules and assumptions,  and included development and procurement costs.
Well-documented: Both estimates were well-documented because  they identified the source of the data, included the work that is  planned, documented the cost-estimating methodologies, showed  how the calculations were made, and were reviewed by management.
Accurate: Both estimates were accurate because they were adjusted  for inflation, based on an assessment of most likely costs for 300  aircraft, and used data from comparable programs where appropriate.
Credible: For both estimates, the Air Force cross-checked major cost  elements to determine whether results were similar and conducted a  risk and uncertainty analysis that included a list of the risks assessed.  For example, the analysis assessed the risk that more work may be  required to extend the service life than currently planned. Also, the Air  Force identified some cost drivers, which are the items that have the  greatest effect on the estimated cost, such as labor rates. While the  Air Force estimates exhibited many characteristics of a high-quality  estimate, they had two shortcomings that lessened the estimates’  credibility. First, the analysis did not include an estimate of the total  costs that may occur if more than 300 aircraft are included in the  programs up to the potential maximum of 650 aircraft. Second, the  estimates were not compared to an independently developed  estimate.
Regarding the first shortcoming, best practices state that a credible cost  estimate should include an assessment of how the cost estimate may  change in response to changes in key program assumptions. This is  known as a sensitivity analysis. Such an analysis helps decision makers  identify areas that could significantly affect a program’s cost, such as  changes in program quantities. According to Air Force officials, the  service may choose to upgrade and extend the service life of additional  aircraft if there are further delays in F-35 production. For example, in a  written statement, Air Force leaders testified in March 2012 that the intent  is to include 350 F-16 aircraft in the upgrade and life extension  programs—50 more aircraft than originally planned. Air Force officials  explained that they did not assess the range of costs that may result from  expanding the number of aircraft in the program up to the maximum of  650 since the program is currently approved for only 300 aircraft.  However, without an assessment of how much the total cost may  increase as the aircraft quantity increases, the estimate lacks  transparency on the full range of possible costs, which could be  significant. For example, we calculated that the cost to upgrade and  extend the service life of all 650 aircraft could total $5.53 billion.
The second major shortcoming is that the Air Force did not compare the  estimates to independently developed cost estimates. Having an  independent estimate is considered one of the best and most reliable  estimate validation methods since an independent estimate is an  objective assessment of whether the program office estimate is  reasonable and can be achieved. Further, our past work has shown that  an independent estimate tends to be higher and more accurate than a  program office estimate. Air Force officials explained that it is likely the  Air Force Cost Analysis Agency will be requested to develop a formal  independent cost estimate for upcoming program reviews, but the  decision to require an independent estimate has not yet been made.  Further, the program review for the capability-upgrade program will not be  held for a year—it is currently scheduled for the fourth quarter of fiscal  year 2013. Until the Air Force obtains an independent cost estimate,  decision makers will not have the assurance that the programs’ estimated  cost can be relied upon for making budgeting and trade-off decisions.  Furthermore, a program that has not been reconciled with an independent  estimate has an increased risk of proceeding underfunded because an  independent estimate provides an objective and unbiased assessment of  whether the program estimate can be achieved.
section_title: Navy Estimate Exhibits Characteristics of a High- Quality Estimate but Does Not Capture Costs of Additional, Nonstructural Work
paragraphs: The Navy’s $2.19 billion cost estimate for the service-life extension of  F/A-18s was comprehensive, well-documented, and accurate. However,  the estimate was not fully credible because it did not include an  assessment of the costs for additional work on the aircraft that may be  done at the same time as the service-life extension and the estimate was  not compared to an independently developed estimate. Since the Navy  will assess on an aircraft-by-aircraft basis whether to do this additional  work, the costs would be in addition to the $2.19 billion for the service-life  extension and most of these costs are not included in the Navy’s  spending plans through fiscal year 2017. In assessing the extent to which  the Navy’s cost estimate exhibited the four characteristics of a high- quality estimate, we found the following:
Comprehensive: The Navy’s estimate identified key cost-estimating  ground rules and assumptions. Also, the Navy’s estimate included the  costs to buy and install the materials needed to repair the airframe so  that it can fly the additional 1,400 hours. Finally, the Navy’s analysis  included assessing the costs for different quantities from 150 up to  280 aircraft.
Well-documented: The estimate was well-documented since it  described the methodology used and the calculations performed to  develop the estimate so that a cost analyst unfamiliar with the  program could replicate it. Also, the Navy’s documentation included  steps to ensure data reliability, and the estimate was reviewed by  management.
Accurate: The Navy’s estimate was accurate since it was adjusted for  inflation, not overly conservative or optimistic, and incorporated data  from comparable programs where appropriate.
Credible: Finally, the Navy’s estimate was partially credible because  the Navy followed some, but not all, of the best practices for this  characteristic. For example, the Navy conducted a sensitivity analysis  which identified cost drivers, such as materiel and labor costs. Also,  the Navy conducted a risk and uncertainty analysis which accounted  for some risks, such as the risk that the service-life extension may  require more work than currently planned. However, the Navy did not  assess the potential range of costs for all work that might be done and  focused only on the costs associated with extending the life of the  airframe, which officials described as consistent with Navy guidance.
Specifically, the Navy told us it will assess on an aircraft-by-aircraft basis  whether the F/A-18s also need capability upgrades to maintain mission  effectiveness, such as adding the ability to integrate with newer aircraft.
According to the Navy, such capability upgrades could cost an average of  about $1.76 million per aircraft, but uncertainty exists about how many  aircraft the Navy will actually decide to upgrade and therefore the total  cost associated with these upgrades is uncertain. In addition, the Navy  has determined that some of the F/A-18 aircraft may also require life  extension or replacement of nonstructural parts or systems that are  becoming obsolete. The Navy estimated that this work could average  $5.64 million per aircraft. Navy officials explained that they will decide on  an aircraft-by-aircraft basis whether to do this work when the aircraft is  inspected just prior to beginning service-life extension. Navy officials told  us that if an aircraft needs a lot of this type of work, the Navy may decide  not to extend the service life of that particular aircraft, but rather substitute  another one in its place. Given the uncertainties, the costs associated  with this additional work are not included in the Navy’s $2.19 billion  estimate nor are they included in the Navy’s spending plans through fiscal  year 2017. The purpose of a sensitivity analysis is to identify the effects  on the estimate of changing key factors and to show a resulting range of  possible costs. Further analysis could show the likelihood of the Navy  incurring some or all of the additional costs. Without an assessment of the  effect on the total costs that would be incurred if some of the 150 aircraft  need additional work at an additional cost, decision makers will not know  the full range of potential costs.
Finally, the Navy did not compare the estimate to an independently  developed cost estimate such as one that could have been requested  from the Naval Center for Cost Analysis. Navy officials stated that the  estimate was reviewed extensively within the Navy and that an  independent estimate was not required because the service-life extension  is a modification of an existing aircraft and is not a separate acquisition  program. Even if not required, Navy officials said that they could have  requested the Naval Center for Cost Analysis to develop an independent  estimate for this program. In our prior work, we have found that an  independent cost estimate is considered to be one of the most reliable  validation methods. If the Navy’s estimate were validated by an  independent cost estimate, decision-makers could place more credibility  in the estimate.
section_title: Conclusions
paragraphs: The Air Force and the Navy decided to upgrade and extend the service  life of selected F-16 and F/A-18 aircraft to provide a capability and  capacity “bridge” until the F-35 is available in sufficient numbers. The cost  to do so is already estimated to total almost $5 billion and likely will  increase due to slippage in the F-35 program and increases in program  scope. In many respects, the Air Force and Navy cost estimates we  evaluated were developed in accordance with some best practices.  Specifically, we found the estimates to be well-documented and accurate  and mostly comprehensive. However, the Air Force and Navy did not  follow some important best practices and as a result did not clearly  identify the potential total costs that may result as the programs evolve.  For example, without an analysis that includes the potential for increases  in the number of aircraft in the program or that clearly identifies the cost  for additional work that may be required on some of the aircraft, decision  makers are not aware of how much total costs could increase if these  events occur. Further, unless the services reconcile the program cost  estimate with an independent cost estimate, there is an increased risk  that the actual costs will exceed the estimate. According to the GAO Cost  Estimating and Assessment Guide, independent estimates are usually  higher and more accurate than program estimates and therefore if a  program estimate is close to an independently developed estimate, one  can be more confident that the estimate is credible. The guide further  explains that there are various types of independent cost estimate  reviews that can be performed, including seeking an independent  estimate. Since these are multibillion-dollar investments that are critical to  maintaining fighter capacity, an independent review of service estimates  or an independently-developed estimate—which could be performed by  the respective service cost-estimating agencies—would provide  assurance about the likely full costs. Without fully credible cost estimates,  service and congressional decision-makers will not have reasonable  confidence of knowing the potential range of costs to upgrade and extend  the service life of selected F-16 and F/A-18 aircraft as they make  resource and trade-off decisions, develop and review budget requests,  and assess the programs’ affordability.
section_title: Recommendations for Executive Action
paragraphs: To improve future updates of Air Force and Navy cost estimates for  upgrading and extending the service life of selected F-16s and F/A-18s  and to improve the ability of decision-makers to assess the potential total  costs, we recommend that the Secretary of Defense take the following  four actions:  Direct the Secretary of the Air Force to update its cost estimates, and in  doing so:  include in its sensitivity analysis an assessment of the range of  possible costs if the capability-upgrade and service-life extension  programs are expanded to more than 300 aircraft including up to the  maximum of 650 aircraft, and obtain an independent review of the updated cost estimates.
Direct the Secretary of the Navy to update its cost estimates and in doing  so:  include in its sensitivity analysis an assessment of the range of  possible costs for extending the service life of other nonstructural  components that are becoming obsolete and capability upgrades that  may be required for some of the 150 aircraft, and obtain an independent review of the updated cost estimates.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOD for comment. In written  comments on the draft of this report, DOD agreed with all four  recommendations.  DOD’s comments are reprinted in their entirety in  appendix III.  DOD also provided technical comments, which we  incorporated into the report as appropriate.
Regarding the first recommendation for improving future updates of the  Air Force cost estimates, DOD concurred, stating that the Air Force had  updated planned force structure requirements to upgrade and extend the  service life of 50 aircraft in addition to the original 300 aircraft and  therefore estimated costs for 350 aircraft.  Also, the department stated  that while these programs could be expanded beyond 350 aircraft, further  cost excursions are premature until F-35 production has matured.  The  department stated that it would update its cost estimates with another  sensitivity analysis containing an assessment of the range of possible  costs if force structure requirements change further.  While we  acknowledge that estimating costs for 350 aircraft is a step in the right  direction, an assessment of how much the total cost may increase up to  the maximum of 650 aircraft is needed to provide transparency on the full  range of costs, which could be significant—potentially totaling up to $5.53  billion.  DOD also noted that further cost excursions are premature until F- 16 structural testing is complete.  However, Air Force officials explained  during our review that data from testing are available throughout the  testing period and that they could use this information to update cost  estimates before testing is completed in 2016.  Until the Air Force  completes analyses to show the potential range of costs, decision makers  will continue to lack crucial data on how much total costs could increase  as they make resource and trade-off decisions.
Regarding the second recommendation for improving future updates of  the Air Force cost estimates by obtaining an independent review of the  updated cost estimates, DOD agreed, stating that the Air Force Cost  Analysis Agency had reviewed the estimates for the Air Force’s fiscal  year 2014 budget development process and that, if further cost  assessments are warranted, the Air Force would submit the estimates for  an independent review.  An independent review of cost estimates is  important because a program that has not been reconciled with an  independent estimate has an increased risk of proceeding underfunded  since an independent estimate provides an objective and unbiased  assessment of whether the program estimate can be achieved.  As the  Air Force updates its cost estimates for these $2.61 billion programs, up- to-date, independently validated cost estimates would provide decision- makers assurance that the programs’ estimated cost can be relied upon  as they assess program affordability.
Regarding the two recommendations for improving future updates of the  Navy cost estimates, DOD concurred but did not explain when or what  specific actions it intends to take to implement these recommendations.   We are encouraged that the department agrees these are important, valid  steps to take and further believe that it is important for the Navy to  implement these recommendations in a timely manner to facilitate review  and implementation of these programs since the programs are estimated  to cost $2.19 billion and could increase an average of $5.64 million per  aircraft.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense, the Secretary of the Air Force, and  the Secretary of the Navy. The report also is available at no charge on the  GAO website at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (404) 679-1816 or pendletonj@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. Staff members making key contributions to  this report are listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To describe the Air Force’s and the Navy’s plans to upgrade and extend  the service life of selected F-16s and F/A-18s, we reviewed the services’  documentation of the programs’ purpose and scope including the  underlying analysis of alternative approaches. To identify the Air Force’s  and the Navy’s estimated costs to upgrade and extend the service life of  F-16s and F/A-18s, we obtained the services’ cost estimates for these  programs and obtained any updates to these estimates. The Air Force  updated its initial estimate to support development of the fiscal year 2013  budget. The Navy’s cost estimate was reported to Congress in May 2011.  All of the cost estimates from these sources were converted to fiscal year  2013 dollars using the indexes published by the Office of the Secretary of  Defense (Comptroller) in the National Defense Budget Estimates for  Fiscal Year 2013, commonly referred to as the “Green Book”.
To assess the extent to which the cost estimates exhibited characteristics  of a high-quality cost estimate, we compared how the estimates were  developed to the cost estimating best practices outlined in the GAO Cost  Estimating and Assessment Guide.  The guide is a compilation of best  practices that federal cost-estimating organizations and industry use to  develop and maintain reliable cost estimates that management can use  for making informed decisions. The guide describes 19 best practices for  developing a comprehensive, well-documented, accurate, and credible  cost estimate—the four characteristics of a high-quality cost estimate. We  analyzed the extent to which the Air Force and Navy cost estimates met  each of these four characteristics and assigned each a rating of not met,   The overall rating  minimally met, partially met, substantially met, or met.for each characteristic was based on the average rating of the best  practices for each characteristic. We also held detailed discussions with  service officials and reviewed program documentation to identify key  factors that could affect the potential total costs such as changes in  aircraft quantities or additional work to maintain aircraft mission  effectiveness that may not have been included in the estimates. We  shared with program officials our cost guide, the criteria against which we  evaluated the program’s estimated cost, as well as our preliminary  findings. When warranted, we updated our analyses on the basis of the  agency response and additional documentation that was provided to us.  Finally, we corroborated our analyses in interviews with service officials  responsible for developing the cost estimates.
We conducted our work at the Office of the Under Secretary of Defense  for Acquisition, Technology and Logistics; Office of the Chief of Staff of  the Air Force, Directorate of Force Application; Ogden Air Logistics  Center at Hill Air Force Base; F-16 System Program Office at Wright- Patterson Air Force Base; Air Combat Command; Air Force Cost Analysis  Agency; Air Force Fleet Viability Board; Office of the Deputy Assistant  Secretary of the Navy for Air Programs; Headquarters United States  Marine Corps; Naval Air Systems Command; and the Naval Center for  Cost Analysis.
We conducted this performance audit from May 2011 to November 2012  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings, and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Air Force Cost Estimates for Upgrading and Extending the Service Life of Selected F-16 Aircraft in Then-Year Dollars
subsections: section_title: Dollars in millions
paragraphs: The Navy estimate was reported as a cost per aircraft in fiscal year 2011 dollars. Therefore, there is  not comparable data for showing the Navy estimate in then-year dollars.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, the following staff members  made key contributions to this report: Patricia W. Lentini, Assistant  Director; Brenda M. Waterfield; Benjamin D. Thompson; Joseph M.  Capuano; Ophelia Robinson; Karen A. Richey; Jennifer K. Echard;  Charles W. Perdue; Michael D. Silver; Michael Shaughnessey; and  Richard S. Powelson.