Defense Acquisitions: DOD Needs to Reassess Joint Cruise Missile Costs before Starting New Production Phase
Released: 2010-10-13T00:00:00

Letter

Background

The JASSM program began in 1995 and was to be an affordable, joint program between the Air Force and the Navy to meet an urgent need with a streamlined acquisition strategy. JASSM predecessor TSSAM was also planned to be a low-cost cruise missile able to deliver several different munitions. However, after several unsuccessful flight tests, the lead contractor for TSSAM initiated a reliability improvement program to address higher reliability requirements, but demonstration of whether problems had been resolved would have taken several years and cost more than $300 million. As costs for TSSAM increased, the Army ended its participation in the program and after a period of declining budgets and changes to threat scenarios, a cost and operational effectiveness analysis was completed, which showed that other options might be adequate to meet national security requirements. In 2004, the Navy left the JASSM program citing it as a redundant capability to other systems in its inventory. JASSM was expected to require minimal maintenance while in storage and life-cycle cost was to be controlled through improved reliability and supportability achieved during development.

To execute the acquisition strategy and meet cost and schedule goals, the Air Force used Total System Performance Responsibility (TSPR). TSPR generally gives the contractor total responsibility for the entire weapon system and for meeting DOD requirements, with minimum government oversight. The Air Force made initial JASSM requirements flexible to allow Lockheed Martin to have clear control of the design and product baseline. Program officials stated this strategy was based on other successful programs, such as the Joint Direct Attack Munition program, and would allow the contractor flexibility to make changes to meet cost and schedule deadlines without having to consult with the government. An example of this flexibility was the mission missile effectiveness requirement. The effectiveness requirement is the minimum number of missiles required to kill specified targets and was named as a key performance parameter, allowing trades between reliability, survivability, and lethality. In other words, if the program was successful at achieving high levels of survivability and lethality, reliability could remain low, even fluctuate, and still meet the stated parameters. Quantities for JASSM were established by reviewing the threshold targets and determining the number of missiles necessary to meet operational damage criteria, based on missile performance using the effectiveness requirement. Therefore, changes to reliability would affect the quantities necessary to meet requirements.

As part of the program’s 1995 acquisition strategy, the Air Force received five proposals for JASSM and in 1996 selected Lockheed Martin and McDonnell Douglas to begin a 24-month risk-reduction phase. Following the risk-reduction phase, the Air Force planned 32 months for development and a total of 56 months from program start to full-rate production in 2001. The program planned for concurrent developmental and operational testing and evaluation with four flight tests planned before initial production. The Air Force planned to have nine fixed-price production lots from 2001 through 2009 totaling 2,400 baseline missiles with an initial program cost estimate of $2.2 billion (fiscal year 2010 dollars).

A former Air Force official who was an early JASSM program manager stated the Air Force accepted Lockheed Martin’s proposal, which included favorable fixed-price contract prices for production lots 1 through 5 with the understanding that the prices would increase after Lot 5. JASSM’s acquisition strategy planned for a 74 percent unit cost increase between Lots 5 and 6. The cost increase between Lots 5 and 6 was to occur at a time when quantities were increasing. Despite this planned cost increase, the production unit costs would have remained within the Air Force’s acceptable range established before the competition and at much less cost than TSSAM. Further, the prices offered by Lockheed Martin for the first five production lots were below the Air Force’s desired cost range for the system. Air Force officials said the low costs contributed to Lockheed Martin’s selection. However, to maintain the benefits of this pricing, the quantities purchased by the Air Force had to remain within a certain range for each of the first 5 years.

While the Air Force planned for some cost growth in the original acquisition strategy, the program’s cost grew much more than expected. For the first four production lots, the Air Force benefited from the favorable prices in the original contract. However, because of funding limitations, it was not able to procure the minimum missile purchase in Lot 5 and had to renegotiate this lot with Lockheed Martin. In doing so, Lockheed Martin was able to renegotiate Lot 5 prices based on its actual production costs—at over $1 million per missile. Air Force documentation indicates that previously negotiated unit prices for Lot 1 through Lot 5 were as much as 45 percent less than Lockheed Martin’s actual costs. Subsequent lots that had not been negotiated under the original contract similarly reflected an increase in price.

Most of this cost growth took place prior to 2006, culminating in a critical Nunn-McCurdy unit cost breach late in 2006. According to program documents, several causes have been cited for the critical Nunn-McCurdy unit cost breach: an unrealistic cost estimate resulting from a flawed acquisition strategy; the addition of 2,500 more expensive JASSM-ER variants; the costly efforts to overcome reliability problems; and reduced annual production rates for a longer period. Following the critical Nunn- McCurdy unit cost breach, the Air Force halted production of the missiles until DOD certified that the program should continue. The Under Secretary of Defense for Acquisition, Technology, and Logistics (USD/ATL) found no lower cost alternatives and certified the program in 2008, despite the missile’s higher than projected production costs.

Since operational testing began in 2001, the reliability of the JASSM missile has been inconsistent. The Air Force flight-tested 62 baseline missiles from January 2001 through May 2007, resulting in 25 failures and 3 “no tests,” which was a 58 percent reliability success rate. However, because the program’s strategy allowed for the contractor to manage to mission effectiveness by combining reliability with other factors, the 58 percent reliability rate was sufficient to meet mission effectiveness criteria. The Air Force tracked reasons for flight test failures, but was not part of the failure review boards until production Lot 5, 5 years after the start of operational testing. Air Force officials stated that until 2006, Lockheed Martin handled all flight test failure review determinations and made corrective actions internally and the government was not heavily involved. During the Nunn-McCurdy certification process, USD/ATL directed the JASSM program to develop a reliability growth plan that would achieve 90 percent reliability for the baseline missile. The program set a goal of achieving this reliability rate by Lot 11, or fiscal year 2013. In addition, the JASSM-ER program set a reliability goal of 85 percent by Lot 4, or fiscal year 2014.

In our 2000 report on JASSM, we recommended the Secretary of Defense revise its acquisition strategy for the JASSM program to be more closely linked to demonstrating that the missile design is stable and can meet performance requirements before making the production decision. DOD partially concurred with our recommendation stating that its acquisition strategy is directly linked to knowledge points, that it is linked to specific criteria established for making the low-rate initial production decision, and that the contractor is required to meet these criteria. We concluded that, while the Air Force had taken steps to link production decisions for JASSM to knowledge, we did not believe that the specific criteria established to support a production decision were sufficient to minimize cost and schedule risk.

In June 2010, the Secretary of Defense announced an initiative to restore affordability and productivity in defense spending. He stated that there is a need to abandon inefficient practices accumulated in a period of budget growth and learn to manage defense dollars in a manner that is “respectful of the American taxpayer at a time of economic and fiscal stress.” He set a goal to save $100 billion over the course of the 5 year defense planning period. Subsequently, USD/ATL has issued guidance on delivering better value to the taxpayer and improving the way DOD does business. That guidance indicated that budget savings could be found by eliminating unneeded and costly programs and activities as well as by conducting needed programs and activities more efficiently, such as by stabilizing production rates. Subsequently, in a September 14, 2010, memorandum, USD/ATL provided specific guidance to acquisition professionals to achieve this mandate. That guidance included 23 principle actions to improve efficiency, including “Mandate affordability as a requirement” and “Drive productivity growth through Will Cost/Should Cost management.”

Corrective Actions Have Led to Improved JASSM Test Results but Operational Effectiveness Is Still to Be Demonstrated

Since 2007, the Air Force has enhanced its oversight of the JASSM program and made significant investments to improve its reliability as directed by USD/ATL. As a result of increased reliability testing and investments in reliability initiatives, the Air Force has identified many of the root causes for flight test failures. Since then, design changes and other corrective actions have improved JASSM baseline’s test results significantly—now demonstrating 85 percent success. The JASSM-ER variant has done well thus far, with no scored failures during the first seven flight tests. However, while JASSM baseline missile reliability has improved, it is not expected to achieve the USD/ATL-required level of 90 percent until 2013, and its operational effectiveness has not yet been demonstrated either through operational testing or use in a combat operation.

Enhanced Oversight and Increased Investments Have Led to Corrective Actions and Improved JASSM Reliability

In 2004, after two back-to-back flight test failures, the Air Force formed a reliability enhancement team to address what it considered the loss of confidence in JASSM’s performance, OSD’s concerns about the program, and budget reductions. The team’s report stated that while JASSM’s development and reliability were within acceptable ranges when compared to other cruise missiles, the JASSM program should increase testing to discover additional weaknesses in design or production as well as increase confidence in the level of reliability achieved and tie those results to contractor incentives. In 2007, after direction from USD/ATL and the Air Force during the Nunn-McCurdy certification process, the program office updated the Joint Reliability and Maintainability Evaluation Team and Test Data Scoring Board charters to significantly expand their role in management of system development, manufacturing, configuration changes, and testing. Since 2007, as a result of the Air Force’s increased attention to reliability testing and investments in reliability initiatives, the program has identified many of the root causes for reliability failures. While there is no single cause behind JASSM flight test failures, common failures occurred across JASSM subsystems including navigation, flight control, and propulsion.

Most of the corrective actions to address the causes of the flight test failures affect missile hardware and many have been implemented in the current configuration for new production missiles. However, some flight test failure investigations are still ongoing. Those investigations are often difficult because of the lack of physical evidence after the flight test missile detonates on the White Sands missile range. As a result, identifying the root causes for failures were based on very extensive component testing at supplier facilities. Additionally, the root causes for several test failures were never conclusively determined as the failures may have resulted from aircraft or user malfunctions. Efforts to address significant reliability problems found during testing have contributed greatly to JASSM’s cost growth and schedule delays since the beginning of development. The Air Force has estimated that it may ultimately spend about $400 million through fiscal year 2025 on its reliability improvement initiatives.

The Air Force has also increased lot acceptance testing of the fuses and implemented high-speed photography and screening improvements. In addition to forensic evaluation of the missile impact area, the Air Force also employs visual inspections and a built-in-test. The Air Force has also taken a variety of actions, in addition to flight testing, to improve JASSM’s reliability, including the following initiatives.

Increased Oversight: The Air Force and Lockheed Martin have begun a process verification program to ensure suppliers follow prime contractor specifications. According to Air Force officials, the process verification program has allowed the Air Force to avoid unforeseen costs as some missile parts have become obsolete. Further, officials stated that it allows the JASSM program to catch problems earlier and plan on how to replace parts sooner. According to a program official, one process verification program team caught an obsolescence issue with a global positioning satellite receiver and was able to minimize the cost and production effect on the program.

Missile Redesign: Program officials state that while wholesale missile redesign is not considered a cost-effective option, they are considering design changes and improvements at the component level.

Increased Personnel: The program office has increased the number of government personnel supporting the process verification program and corrective action efforts. During Lot 1, the program had two staff members with production and manufacturing engineering expertise— by Lot 7, 22 staff members had such expertise.

Improved Quality Assurance: In August 2006, the Air Force and Lockheed Martin implemented a quality assurance program. Lockheed Martin has implemented tests and improvement programs to increase user confidence in reliability and control costs. For example, Lockheed Martin officials stated that, to improve reliability, they have begun using a test that exposes electrical connections to higher voltages than they usually encounter during flight to make sure the wiring can handle a surge. Additionally, Lockheed Martin has increased the sample sizes of certain components they inspect and test.

Recent JASSM Testing Has Shown Improved Results

Recent tests of JASSM have demonstrated increased reliability. Since the Air Force’s reliability initiatives began in fiscal year 2007, the JASSM program has conducted 48 missile flight tests and 39 have been successful (2 were characterized as “no-test”) for a reliability rate of 85 percent.

The current focus of JASSM baseline testing has been on improving the reliability of the missile. In the most recent tests of the JASSM baseline missiles produced in 2008, 15 of 16 flight tests were considered successful. In the one failure, the warhead did not detonate and the program is awaiting fuse recovery to make a determination of the root cause. The JASSM-ER is in developmental testing. Developmental testing of JASSM- ER is primarily addressing the differences of JASSM-ER from the baseline system (i.e., larger engine and fuel tanks) and will verify integration on the B-1 aircraft. All seven test flights of JASSM-ER have been successful. The program office is planning three additional integrated JASSM-ER tests to be flown before a production decision is made.

Uncertain If Improved Missile Reliability Will Result in Improved JASSM Operational Effectiveness

In 2007, after USD/ATL’s decision to enhance JASSM reliability, the Air Force and Lockheed Martin agreed to focus on the inherent reliability of the missile and not take into account user error or platform malfunctions (i.e., carrier aircraft, aircrew instrumentation, range safety, etc.). Whereas operational testing is designed to evaluate the ability of JASSM to execute a mission, reliability testing is more narrowly focused on evaluating the missile’s performance during the mission. While mission failures were counted against the program during initial testing, more recent mission test failures have been declared “no tests.” For example, in early testing when a B-52 software issue resulted in an aborted mission and it was scored a test failure, this event would have been declared a no test under current missile reliability definitions.

While recent flight testing of the baseline missile has shown improved missile reliability, the Air Force has not yet evaluated the operational effectiveness and suitability of the baseline JASSM with all corrective actions implemented. The JASSM program assesses operational effectiveness through operational testing, follow-on testing, and the weapon system evaluation program (routine tests of inventory assets). These flight test scenarios assess operational effectiveness in realistic combat scenarios against targets by determining reliability, evaluating capability and limitations by identifying deficiencies, and recommending corrective actions. In operational testing, the JASSM baseline program flight tested 38 missiles from June 2002 through May 2007 resulting in 19 failures and two no tests. While these tests identified issues with missile reliability, they also identified issues related to the B-52 aircraft, aircraft software, and fuse issues which negatively affected the operational effectiveness of the missile. This led to a 9-month suspension of testing in 2004 to address these issues. The improved JASSM baseline missile’s suitability was assessed by Air Force testers in 2008 and it was characterized as suitable and likely to meet reliability goals; however, operational testing of the effectiveness of the improved missile has not yet been scheduled.

Several Areas of Potential Cost Risk Could Further Increase JASSM Program Costs

Current projections of JASSM costs have increased by over 7 percent since the Nunn-McCurdy certification in 2008. When taking into consideration the pre-2008 cost growth, which included the cost of adding the JASSM-ER variant, JASSM has grown from a $2.2 billion to a $7.1 billion program. In addition, while it has initiated several cost control measures, the Air Force appears to have limited options to reduce JASSM costs. Moreover, several areas of risk could add to those costs. First, the Air Force has not been able to provide enough annual funding to support the annual procurement levels used as the basis for its 2008 program cost estimate. That has led to a less efficient production process and a longer production period (most recently extended 5 years to 2025). Second, until the Air Force evaluates the effectiveness of the inventory JASSM baseline missiles with corrective actions for previously identified hardware and software issues, their viability and military utility is in question. If inventory missiles are found not to have utility, they may need to be replaced. If retrofitted missiles are found to be effective, the Air Force may still have to find additional funding to complete the retrofit process. Third, the Air Force plans to conduct many more flight tests to improve JASSM reliability from 85 to 90 percent. Finally, in comparing the capabilities and cost of JASSM to several domestic and international missile systems in 2008, the Air Force assumed that JASSM would cost about $1 million per unit, which is about 40 percent less than currently expected.

JASSM Costs Are Much Higher Than Originally Anticipated

Compared to original program estimates, JASSM’s currently projected costs are much higher because of (1) higher than anticipated production costs, (2) longer production period, (3) the addition of the JASSM-ER variant, and (4) reliability improvement efforts. Through fiscal year 2010, about 75 percent of the planned JASSM quantities have yet to be procured and, as a result, most of the program costs have yet to be incurred.

Following the critical Nunn-McCurdy unit cost breach, the Air Force halted production of the missiles until USD/ATL certified that the program should continue. USD/ATL found no lower cost alternatives and certified the program in 2008, despite the missile’s higher than projected production costs. As a part of our review, we examined the cost estimates used by OSD to certify the program following the critical Nunn-McCurdy unit cost breach. This estimate used the actual costs of the missile since JASSM was well into the production phase at the time. Overall, the Air Force’s cost estimate substantially met our best practice standards in our Cost Guide.

For a more in-depth discussion of our review of this JASSM cost estimate, see appendix III.

Since the Nunn-McCurdy certification in 2008, the growth in JASSM’s projected program cost has moderated, rising about $500 million (from $6.6 billion to $7.1 billion) through 2025. Reliability enhancements to the JASSM missile instituted in 2007 and additional reliability testing have added the majority of the increase in program costs. These enhancements were implemented to meet USD/ATL’s 90 percent reliability goal which was set during the Nunn-McCurdy certification. Also, the Air Force decided to lengthen the program’s procurement schedule by another 5 years, buying the same number of missiles over a longer time period. That reduces the efficiency of the production processes and adds inflation to the cost estimate. Currently, on a per unit basis, the average procurement unit cost of a JASSM missile is projected to be about $1.2 million. JASSM- ER is expected to cost about $200,000 more than the average, about $1.4 million per unit.

Since 2008, the Air Force has added several measures to control costs in the JASSM program, but the effect of these measures is not yet clear. Examples of these measures include the following.

Contract Incentives: The Air Force has begun using fixed-price incentive (firm target) contracts for each lot to produce the missiles for less than projected.

Greater Insight into Actual Costs: Air Force officials have increased insight into Lockheed Martin’s actual costs, which may make them more informed when negotiating new contracts. Program officials stated that, for example, they now know how many engineers are needed to perform a certain task and the number of hours it takes to assemble a missile. The Air Force can directly verify the costs charged by subcontractors.

Increased Authority over Design: In recent contract negotiations, the Air Force gained approval authority over certain design changes that may affect current and future lots, including those that may increase cost, require retrofit, or affect safety. Previously, Lockheed Martin had full authority over most design changes.

While the effectiveness of the cost control measures is not yet known, the Air Force appears to have limited options to actually reduce those costs. For example, annual production rates are expected to remain well below the levels projected at the start of the program. The 2008 Air Force cost estimate was based on an annual production rate of 280 missiles per year. However, that cost estimate may now be understated because the program has not produced that many missiles in a single year since 2005. For example, the Air Force’s procurement quantities for production Lot 7 and Lot 8 were 111 and 80 units, respectively, well below the economic order quantity of 175 missiles per year. Program officials stated that annual quantities below the economic order quantity will result in an increasingly inefficient production process and some key suppliers may shift from continuous to limited production. Further, Lockheed Martin officials stated that low production rates could cause skilled labor to look elsewhere for work and JASSM reliability could be adversely affected. The contractor has been able to maintain some level of production efficiency because of foreign military sales that make up for the reduced Air Force procurements. However, lower than projected annual procurement levels will increase production costs. A further challenge is the fact that JASSM’s design is mostly complete and there may be few opportunities to reduce production costs through redesign. As a result, average JASSM unit costs may remain in excess of $1.2 million indefinitely.

Effectiveness of Retrofitting Corrective Actions into JASSM Inventory Is Uncertain

The Air Force has plans to address the low reliability of missiles in its inventory by retrofitting some of its 942 missiles with hardware and software corrective actions. Program officials state the retrofit costs will be shared between Lockheed Martin and the Air Force, but the total cost to retrofit the missiles in inventory has not been calculated. However, previous efforts to retrofit JASSM missiles have proven to be problematic. An example of challenges associated with retrofitting missiles is adding telemetry instrumentation kits after the missiles have been produced and are in the inventory. Those kits are added to all missiles to be flight tested. This requires opening up the missile to insert telemetry after the stealth coating has been applied and increasing the number of electrical connections as compared to a production missile. Air Force officials stated the kit could add some reliability concerns when it is added to test missiles because the missiles were not designed to be opened after they were completed. Air Force officials also stated that workers have to reroute wires and remove the engine so that the self-destruct mechanism can be installed and all of this rework inside the missile has the potential to lead to more errors and cause additional reliability issues.

The impact of retrofitting missiles has become evident in the weapon system evaluation program, which is operationally representative flight testing run by users of the system and focuses on the performance of missiles in the inventory. JASSM’s performance in this evaluation program has not been good, with 7 failures in 12 tests from 2006 through 2007, and with at least some of the failures attributable to the retrofit process. The addition of telemetry kits has also contributed to 3 no tests during other JASSM flight testing. The Air Force has not yet flight tested any of the JASSM inventory missiles that have been retrofitted with all of the corrective actions to address reliability issues. This type of test would be important in determining the viability of the current inventory of JASSM missiles and would be a key input in the Air Force deciding whether or not to retrofit the entire inventory of missiles.

Additional Flight Testing to Further Improve Reliability Will Be Costly

The Air Force plans more flight tests in the next few years of new production missiles to meet missile reliability goals. For the JASSM baseline missile to meet its reliability requirement, the Air Force is planning to conduct up to 48 additional flight tests, at a cost of about $120 million. In addition, most reliability issues with the baseline variant will directly affect the progress of the JASSM-ER variant as the missiles are at least 70 percent common in hardware and 95 percent common in software. Anything learned during these flight tests about the baseline applies to JASSM-ER. According to the Air Force, as many as 20 additional flight tests may be needed to fully demonstrate JASSM-ER’s reliability goal of 85 percent.

The $190 million cost to achieve the final percentages of missile reliability reflects the fact that problems or weaknesses become harder to find and correct as the more obvious issues are corrected. Program officials are considering alternative means to meet user needs for a more reliable missile while reducing the cost of JASSM flight testing.

Previous Analysis of Alternatives Assumed Much Lower Costs of JASSM Missile

As part of the Nunn-McCurdy certification process in fiscal years 2007 through 2008, DOD assessed whether there were readily available alternatives that provided as much or more capability as JASSM at lower cost. DOD assessed programs ranging from direct attack munitions to intercontinental range missiles. For the JASSM baseline missile, all of DOD’s existing programs were found to be less effective in terms of lethality, survivability, or capacity. The Navy’s Tomahawk missile was the closest alternative to meeting JASSM’s capability but it is not as lethal as JASSM. Also, the Tomahawk is launched from ships and not from aircraft, as the Air Force plans to use the capability.

DOD also evaluated new or modified programs as possible alternatives to JASSM and JASSM-ER. The Air Force evaluated 12 domestic and international missile systems with projected unit production costs ranging from $600,000 to $2.8 million. JASSM-ER, estimated at the time of this evaluation to cost $1 million per unit, was more expensive than 5 alternative systems under consideration. In terms of performance, some alternatives were more capable than JASSM and some were not. All of the alternative systems were expected to require some up-front investment. Based on this analysis, no other alternative was found to provide greater or equal military capability at less cost than JASSM-ER.

Later in the Nunn-McCurdy process, however, OSD cost analysts found that the costs of JASSM-ER would likely be at least $1.4 million per missile. That continues to be the projected unit cost of JASSM-ER and, as we discussed earlier in this report, there are cost risks that may drive that unit cost higher. Despite the higher production unit costs for JASSM-ER, the Air Force has not revisited the results of its assessment of alternatives. In light of the current cost projections, which are 40 percent higher than assumed in the previous assessment, JASSM-ER would be equal to the cost of an additional alternative. Further, the unit cost differential between JASSM-ER and the lower-priced alternatives may now be large enough to make those alternatives more competitive in terms of cost or capabilities.

Conclusions

DOD’s 25-year history to acquire and field an affordable air-to-ground cruise missile has been a difficult one. After abandoning the expensive TSSAM program, the Air Force conceived the JASSM program using an acquisition strategy that minimized government oversight. After restructuring the program in 2008 and after considerable effort to improve reliability, the JASSM program as it exists today is much different than originally envisioned. A $2.2 billion, 11-year program to produce 2,400 missiles has become a $7.1 billion, 28-year program to produce 4,900 missiles. From a technical and capability standpoint, the program offers more now than the baseline missile did before 2008. Yet, the effectiveness of the new missiles remains to be demonstrated in operational testing, and low production rates, retrofit costs, and additional reliability testing could drive program costs higher.

At this point, about 70 percent of the projected JASSM costs have not yet been incurred. In November 2011, DOD will decide whether to approve the Air Force’s request to start low-rate initial production of the JASSM-ER variant. Low-rate initial production is normally the last major milestone decision for an acquisition program. With the JASSM program now expected to extend through 2025 and about $5 billion yet to be spent, a reevaluation of its cost-effectiveness is warranted before such a commitment is made. This is particularly true given the Secretary of Defense’s recent initiative to improve the cost-efficiency of defense acquisition programs. At this juncture, the JASSM program would seem to be an excellent opportunity for DOD and Air Force leadership to take a hard look at the cost-effectiveness and efficiency of this important but costly defense program.

Recommendation for Executive Action

We recommend that the Secretary of Defense defer the production decision for JASSM-ER until (1) the program’s likely costs and affordability are reassessed to take into account the feasibility and cost of retrofitting JASSM baseline missiles or replacing them, the cost of additional reliability testing against the likely improvement, and the effect of sustained low production rates; and (2) the results of the previous analysis of alternatives are reassessed in light of the likely costs of the JASSM program.

Agency Comments and Our Evaluation

In its comments on our draft report, DOD partially concurred with our recommendation. DOD stated that JASSM-ER is on track for a Milestone C low-rate initial production decision in November 2010. DOD also agreed that the rate of JASSM production has not been optimum and that it plans to address efficient production rates as part of the JASSM-ER Milestone C decision. DOD also stated that (1) there are no additional plans (nor is there a need) to retrofit fielded JASSMs above what has already been accomplished or is under way; (2) it has revisited various alternatives and reaffirms the continued validity of its 2008 conclusion that none of the alternative concepts provide comparable operational utility at or near a similar cost or schedule to JASSM; and (3) in the absence of viable alternatives, delaying the program further will increase costs and further postpone delivering a vital capability to the warfighter. DOD’s response is reprinted in appendix II.

In concluding that retrofits to the inventory missiles may not be necessary, DOD does not address the viability of the current inventory of JASSM baseline missiles or the need to replace some or all of them. Until the Air Force evaluates the effectiveness of the inventory of JASSM baseline missiles with corrective actions for previously identified hardware and software issues, their viability and military utility will still be in question. In addition, DOD states that the Air Force has revisited its earlier assessment of alternatives to JASSM and again found that there are none with comparable utility, cost, or schedule. This is new information and DOD did not provide details for us to assess, including whether the Air Force factored in the higher current projections of JASSM costs. Finally, DOD did not address the part of our recommendation dealing with the cost of additional reliability testing against the likely improvement.

To the extent DOD has made decisions on retrofits and reconsideration of alternatives, these are positive signs, as is its agreement to address the efficiency of production rates. At this point, it is not clear whether the reliability of the existing baseline inventory missiles is acceptable or whether additional reliability testing is warranted. These determinations are necessary to establish the full value and cost of the JASSM program.

Beyond these steps, it is incumbent upon the department to reexamine JASSM before making the production decision to ensure that the program is structured as efficiently as possible and is still a good investment given the other demands DOD faces. DOD’s agreement to address the efficiency of JASSM production rates is a positive step. This is particularly important given the Secretary’s current efficiency and affordability initiative. DOD needs to ensure that it has the information available to fully assess the JASSM investment before making the production decision. If DOD needs more time, then we believe the decision could be delayed.

We also received several technical comments from DOD and the Air Force and have made other changes to our report.

We are sending copies of this report to the Secretary of Defense and interested congressional committees. In addition, this report will be made available at no charge on the GAO Web site at http://www.gao.gov. If you or your staff have any questions about this report or need additional information, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

To determine Joint Air-to-Surface Standoff Missile’s (JASSM) current production unit costs and the extent they have grown, we analyzed JASSM’s contracts, budgets, and compared the program’s selected acquisition reports. We analyzed Nunn-McCurdy documentation and certification criteria as well as Air Force and Lockheed Martin data to determine the causes for the cost growth and critical breaches. We interviewed officials with the JASSM joint program office; Lockheed Martin; the Office of Secretary of Defense Cost Assessment and Program Evaluation; and a former program official.

To determine the extent to which Department of Defense’s (DOD) cost estimating policies and guidance support the development of high-quality cost estimates, we analyzed the cost estimating practices of the Cost Analysis Improvement Group (CAIG), now known as the Cost Analysis and Process Evaluation (CAPE), in the development of life-cycle cost estimates for the Air Force’s Joint Air-to-Surface Standoff Missile Program (baseline and JASSM-ER variants), against the 12 best practices of a high- quality cost estimate as defined in our Cost Estimating and Assessment Guide.

We assessed each cost estimate, used in support of the critical Nunn- McCurdy unit cost breach, against these 12 key practices associated with four characteristics of a reliable estimate. As defined in the guide, these four characteristics are comprehensive, well-documented, accurate, and credible, and the practices address, for example, the methodologies, assumptions, and source data used. We also interviewed program officials responsible for the cost estimate about the estimate’s derivation. We then characterized the extent to which each of the four characteristics was met; that is, we rated each characteristic as being either Not Met, Minimally Met, Partially Met, Substantially Met, or Fully Met. To do so, we scored each of the 12 individual key practices associated with the four characteristics on a scale of 1-5 (Not Met = 1, Minimally Met = 2, Partially Met = 3, Substantially Met = 4, and Fully Met = 5), and then averaged the individual practice scores associated with a given characteristic to determine the score for that characteristic.

To determine the results of the most recent tests and if corrective actions have been implemented for previous test failures, we analyzed JASSM flight test results and failure review board findings to determine scoring criteria and results to determine what corrective actions were implemented. We determined whether recent flight test results are representative of the entire fleet by comparing and evaluating the lot-by- lot missile configuration changes and retrofit activities. We interviewed officials with the JASSM joint program office; Lockheed Martin; Office of Under Secretary of Defense for Acquisition, Technology, and Logistics; Director, Operational Test and Evaluation; Air Combat Command; Secretary of the Air Force for Acquisition; Office of Secretary of Defense Cost Assessment and Program Evaluation; Joint Staff; Air Force Directorate of Test and Evaluation; and a former Air Force official who was an early JASSM program manager to better understand the program’s objectives and original acquisition strategy. We discussed recent DOD reliability initiatives with Office of the Director, Operational Test and Evaluation officials.

To determine what the Air Force has done to control and reduce production costs while improving reliability, we examined JASSM’s contracts to see what provisions have been added as well as Air Force and Lockheed Martin data. We interviewed officials with the JASSM joint program office; Lockheed Martin; and Office of Secretary of Defense (OSD) test organizations to determine if testing reflects the current effectiveness of the missile. We reviewed our prior work on best practices for a knowledge-based approach for acquisition programs in determining if the Air Force’s approach to beginning the JASSM-ER program meets best practices. We compared requirements and other documents to see if the JASSM-ER missile reflects lessons learned from the baseline variant as well as increased knowledge and oversight from the government. We compared the baseline design with JASSM-ER to determine commonality. We interviewed officials with the JASSM joint program office; Lockheed Martin; Office of Under Secretary of Defense for Acquisition, Technology, and Logistics; Office of Secretary of Defense Cost Assessment and Program Evaluation; Director of Operational Test and Evaluation; Air Combat Command; Secretary of the Air Force for Acquisition; Joint Staff; and a former program official to the determine acquisition planning leading up to JASSM-ER’s production decision and what initiatives have been taken to control costs.

Appendix II: Comments from the Department of Defense

Appendix III: JASSM Baseline and JASSM-ER Cost Estimates Met Most Best Practices, but the Risk Analyses Did Not Consider Reliability or Extending the Production Schedule

After reviewing documentation submitted by the JASSM program office, conducting interviews, and reviewing relevant sources, we determined the CAIG’s life-cycle cost estimate’s totaling $7.1 billion for both programs— the JASSM baseline cost estimate was $3.4 million while the JASSM-ER variant cost estimate of $3.7 million—Fully Met one and Substantially Met the other three characteristics of a reliable cost estimate, as shown in Table 3 below. We assessed 12 measures consistently applied by cost estimating organizations throughout the federal government and industry and considered best practices for the development of reliable cost estimates. We analyzed the cost estimating practices used by CAIG in developing the life-cycle cost estimates for both programs against these 12 best practices and the findings are documented in table 3 below.

The following explains the definitions we used in assessing CAIG’s cost estimating methods used in support of the critical Nunn-McCurdy unit cost breach: Fully Met—JASSM program office provided complete evidence that satisfies the entire criterion; Substantially Met—JASSM program office provided evidence that satisfies a large portion of the criterion; Partially Met—JASSM program office provided evidence that satisfies about half of the criterion; Minimally Met—JASSM program office provided evidence that satisfies a small portion of the criterion; and Not Met—JASSM program office provided no evidence that satisfies any of the criterion.

The sections that follow highlight the key findings of our assessment.

JASSM Baseline and JASSM-ER Estimates Substantially Met Characteristics for Comprehensiveness

Though the cost estimates accounted for all possible costs and were structured in such a manner that would ensure that cost elements were omitted or double-counted, neither the JASSM baseline nor JASSM-ER had a Work Break Down Structure (WBS) dictionary that defined each element. In addition, the JASSM baseline variant provided no evidence that risks associated with the ground rules and assumptions were traced back to specific cost elements.

All applicable costs including government and contractor costs were included in the estimates—The cost estimates included sunk costs such as contractor program management, overhead, system design, and development and testing. In addition, the program office outlined the cost estimating methodology, basis of the costs, as well as development costs for JASSM-ER and other government costs.

The cost estimates’ level of detail ensure that no costs were omitted or double-counted—The cost estimates are based on a product-oriented WBS which is in line with best practices. For example, the cost estimate is broken down into various components such as the propulsion, payload, airframe, and guidance and control and also includes supporting cost elements such as systems engineering, program management, and system test and evaluation. As a result, all of the system products are visible at lower levels of WBS providing us with confidence that no costs were omitted or double- counted. WBS has been updated as the JASSM baseline and JASSM-ER programs have evolved; however, there is not an accompanying dictionary that defines each element and how it relates to others in the hierarchy.

Ground rules and assumptions were largely identified and documented—The JASSM baseline cost estimate documentation included a list of risk model inputs based on WBS elements. Although WBS elements such as engineering support, subcontractor, and warranty were identified, there was no discussion of risk upon assumptions that drive costs such as product reliability, sustainability of subcontractors, or schedule variability. Like the JASSM baseline cost estimate documentation, the JASSM-ER cost estimate documentation also included a list of ground rules and assumptions; however, there was evidence that risk associated with the fuel tank assumption was traceable to a specific WBS element. In separate documentation, we were able to identify where the program office considered risks for the JASSM baseline estimate.

JASSM Baseline and JASSM-ER Estimates Fully Met Characteristics for Being Well Documented

Both cost estimates were documented in enough detail that would allow an analyst unfamiliar with the program to recreate the estimate and get the same result. In addition, the briefing to management was detailed enough to show that the estimates were credible and well documented.

The cost estimate is fully documented—For the JASSM baseline and JASSM-ER, the cost estimate documentation included a report documentation page identifying the report date, title, contract number, report authors, and other information. The documentation also included a table of contents, introduction, purpose, and structure of the document as well as the scope of the estimate, a list of team members, the cost methodology, and a system description. The documentation discussed a risk and sensitivity analysis, costs broken out by WBS elements including data sources and estimating method and rationale, and provided evidence that the estimates were updated using actual costs. In a separate briefing, the program office outlined the cost estimating methodology, basis of the costs, as well as development costs for JASSM-ER and other government costs. The program office also provided a copy of the cost sufficiency review of the estimate, which included the estimate’s purpose and scope, technical description and schedule, ground rules and assumptions, data sources and analysis, and methodology. For both programs, the estimate documentation and the cost analysis requirements document (CARD) addressed best practices and the 12 steps of a high-quality estimate. Contingency reserves and the associated level of confidence for the risk-adjusted cost estimate were also documented. Electronic versions of the cost estimates were also provided.

The estimate documentation describes how the estimate was derived—The point estimate was developed primarily using actual costs, with a few cost elements estimated based on learning curves method. Actual sunk costs for prior years were presented and remaining production lot costs were based on a labor staffing assessment and the latest contractor labor rates. Cross-checks were performed and no instances of double-counting were visible. A separate document was provided that showed in detail how the cost estimate was developed, what data were used to create the cost estimate, and how risks were quantified to determine a level of confidence in the cost estimate.

The estimates were reviewed and approved by management— The estimates were presented by OSD CAIG to the OSD overarching integrated product team for consideration as the new acquisition program baseline. In November 2009, the team provided a detailed overview of the JASSM program which addressed the major cost growth factors, such as the addition of the JASSM-ER variant, reliability enhancements, and the reduction in missile purchases.

JASSM Baseline and JASSM-ER Variant Estimates Substantially Met Characteristics for Accuracy

Both cost estimates were unbiased and represented most likely costs. For example, the estimates were adjusted to reflect risks and the program office also included requirements in the new Lot 8 contract that would allow them to update the cost estimates with actual data.

The cost estimates were adjusted for inflation—The JASSM program office used the February 2009 version of the OSD inflation rates provided by the Secretary of the Air Force/Financial Management Cost and Economics. The estimates were developed and documented in base year 1995 dollars and inflated using the weighted rates applicable to the appropriations in the estimate. Base year 1995 is the program’s designated base year.

The cost estimates included most likely costs—Per the Nunn- McCurdy certification process, the CAIG developed independent cost estimates for the JASSM baseline and JASSM-ER development and procurement costs as well as future-year resource requirements for the baseline and JASSM-ER variants. Operating and support costs as well as software costs were also included in the estimates. The JASSM baseline life-cycle cost estimate of $3.4 billion, which spans a period of time from 2001 through 2015, was estimated at the 77 percent confidence level, while the JASSM-ER life-cycle cost estimate of $3.7 billion spans the period from 2011 through 2025 and was estimated at the 73 percent confidence level.

The cost estimates have not been updated to reflect current costs—Though the JASSM baseline estimate dated April 2008 was updated to reflect new program changes, the CARD has not been updated since May 2003. Examples of JASSM baseline changes include additional reliability enhancement team improvements and additional testing, which are not reflected in the 2003 CARD. However, when comparing the JASSM WBS dated January 1999 and the Lot 8 contract dated January 2010, it is evident that WBS has been updated as changes have occurred. On the other hand, the CARD for the JASSM-ER was updated as of August 2009. Updates to the JASSM-ER CARD include a new, more powerful engine than the baseline variant. As part of the Milestone C process, work is currently under way by CAPE to update the JASSM-ER cost estimate. The program office said that the JASSM- ER cost model will include updated costs based on the Lot 8 proposal data, updated quantity profiles, and January 2010 revised inflation rates. The program office is in the process of updating labor rates and overhead rates and is reexamining all component prices.

JASSM Baseline and JASSM-ER Estimates Substantially Met Criteria for Credibility

While the cost estimates addressed risk and uncertainty as well as sensitivity, the estimates failed to address the risks regarding reliability and changes to the production schedule. By not doing so, the program office may not have a full understanding of the future effects to the overall cost position of these two programs.

The estimates were assessed for risk and uncertainty—Both programs identified engineering and test support, subcontractors, and warranty as major risk elements. However, the analysis did not identify reliability or an increase in the production schedule as possible risk factors. During the Nunn-McCurdy certification process, the DOD’s analyses found that the cost breach was driven by four primary factors, two of which focused on reliability. As a result, the program office instituted a reliability enhancement program directed to address reliability concerns. An indirect effect of the enhancement program was an increase in the overall missile costs. The December 2009 selected acquisition report identified increases to the missile hardware cost due to reduced annual quantities, missile production breaks, and increased test requirements and reliability programs. The combined cost estimate, for the JASSM baseline and JASSM-ER variants, has grown significantly over time. By not including reliability and the extension of the production schedule as possible risk factors, cost growth could continue to occur in future production lots. As a result, the programs’ calculated point estimate confidence level of 77 percent for baseline and 73 percent for the JASSM and JASSM-ER variants may be overstated.

The estimates were assessed for sensitivity—For both the JASSM baseline and JASSM-ER estimates, key cost drivers were identified. The cost estimators examined eight cost factors for the JASSM baseline estimate and 13 cost factors for the JASSM-ER estimate. For the JASSM analysis, engineering support, testing support, other subcontractors, and Teledyne propulsion had the greatest impact on the total variance in the estimate. Engineering support showed a 14 percent impact, followed by test support with a 13 percent impact, other subcontractors with a 12 percent impact, and Teledyne propulsion with a 9 percent impact. These four elements account for 68 percent of the total cost before risk was applied. For the JASSM-ER analysis, the Williams propulsion, other subcontractors, engineering support, and testing support showed the greatest impact on the total cost variance in the estimate. The Williams propulsion had a 15 percent impact, followed by a 15 percent impact for other subcontractors, an 11 percent impact for engineering support, and a 9 percent impact for test support. These four elements account for 76 percent of the total cost before risk was applied.

The cost estimates were checked for errors—Cross-checks were performed and no instances of double-counting were visible. The Lot 5 and Lot 6 estimates were compared back to Lot 1 through Lot 4 for consistency and reasonableness. Also, multiple row and column summation cross- checks were performed to avoid duplication and omission errors. Upon review of the electronic cost model, GAO found no instances of double- counting and the spreadsheet calculations are accurate given the input parameters and assumptions.

The cost estimates were validated against an independent cost estimate—The CAIG estimate is the independent cost estimate. As part of the Nunn-McCurdy certification process, the CAIG developed an independent cost estimate for the development and procurement costs as well as future-year resource requirements for the baseline and JASSM-ER variants. This new independent estimate was a joint effort by the OSD CAIG, the program office, and the Financial Management Center of Expertise, so there was no other estimate for comparison. Per the Nunn- McCurdy JASSM certification package dated April 30, 2008, the CAIG estimate of the acquisition costs for the restructured JASSM program is $7.1 billion, which is directly comparable to the $6 billion estimate reported in the quarterly selected acquisition report dated December 2007.

Appendix IV: GAO Contact and Staff Acknowledgments

Michael J. Sullivan, (202) 512-4841 or sullivanm@gao.gov.

In addition to the contact name above, the following individuals made key contributions to this report: William Graveline (Assistant Director), John Crawford, Morgan DelaneyRamaker, Tisha Derricotte, Michael J. Hesse, Karen Richey, Hai Tran, and Alyssa Weir.