Defense Acquisitions: Many Analyses of Alternatives Have Not Provided a Robust Assessment of Weapon System Options
Published: 2009-09-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Department of Defense (DOD) weapon programs often experience significant cost and schedule problems because they are allowed to start with too many technical unknowns and not enough knowledge about the development and production risks they entail. GAO was asked to review the department's Analysis of Alternatives (AOA) process--a key first step in the acquisition process intended to assess the operational effectiveness, costs, and risks of alternative weapon system solutions for addressing a validated warfighting need. This report (1) examines whether AOAs have been effective in identifying the most promising options and providing a sound rationale for weapon program initiation, (2) determines what factors have affected the scope and quality of AOAs, and (3) assesses whether recent DOD policy changes will enhance the effectiveness of AOAs. To meet these objectives, GAO efforts included collecting information on AOAs from 32 major defense acquisition programs, reviewing guidance and other documents, and interviewing subject matter experts.
section_title: What GAO Found
paragraphs: Although an AOA is just one of several inputs required to initiate a weapon system program, a robust AOA can be a key element to ensure that new programs have a sound, executable business case. Many of the AOAs that GAO reviewed did not effectively consider a broad range of alternatives for addressing a warfighting need or assess technical and other risks associated with each alternative. For example, the AOA for the Future Combat System program, one of DOD's large and most complex development efforts, analyzed the operational performance and cost of its alternatives but failed to compare the technical feasibility and risks, assuming that the technologies would perform as forecasted. Without a sufficient comparison of alternatives and focus on technical and other risks, AOAs may identify solutions that are not feasible and decision makers may approve programs based on limited knowledge. While many factors can affect cost and schedule outcomes, we found that programs that had a limited assessment of alternatives tended to have poorer outcomes than those that had more robust AOAs. The narrow scope and limited risk analyses in AOAs can be attributed in part to program sponsors choosing a solution too early in the process, the compressed timeframes that AOAs are conducted under, and the lack of guidance for conducting AOAs. While AOAs are supposed to provide a reliable and objective assessment of viable weapon solutions, we found that service sponsors sometimes identify a preferred solution or a narrow range of solutions early on, before an AOA is conducted. The timing of AOAs has also been problematic. Some AOAs are conducted under compressed timeframes in order to meet a planned milestone or weapon system fielding date and are conducted concurrently with other key activities required to become a program of record. This can short-change a comprehensive assessment of risks and preclude effective cost, schedule, and performance trade offs from taking place prior to beginning development. Furthermore, while DOD has an opportunity to influence the scope and quality of AOAs, it has not always provided guidance for conducting individual AOAs. Recognizing the need for more discipline in weapon systems acquisition, DOD recently revised its overall acquisition and requirements policies. If implemented properly, the revised policies could provide a better foundation for planning and starting new programs with sound, knowledge-based business cases. Included in the revised acquisition policy are several mechanisms to improve the AOA process. For example, the policy revisions should help ensure that DOD direction is provided before AOAs are started and that they are conducted at an early point in the acquisition process where their results can inform decisions affecting program initiation. While these policy changes are promising, DOD must ensure that they are consistently implemented and reflected in decisions on individual programs. Furthermore, more specific criteria and guidance for how AOAs should be conducted may need to be developed to ensure they meet their intended objectives and provide an in-depth assessment of alternatives.

section_title: Letter
paragraphs: Cost, schedule, and performance problems in the Department of Defense’s  (DOD) weapon system programs are serious. Recently, we reported that  the department’s 2008 portfolio of 96 major defense acquisition programs  experienced cost growth of $296 billion, experienced an average delay in  delivering initial capabilities of 22 months, and have delivered fewer  quantities and capabilities to the warfighter than originally planned. Over  the past several years, our work has highlighted a number of underlying  causes for why poor outcomes have occurred in weapon programs. One  key cause is that DOD allows programs to begin without a sound match  between requirements and the resources needed to achieve them. That is,  programs enter the acquisition process with requirements that are not fully  understood, cost and schedule estimates that are based on optimistic  assumptions, and a lack of sufficient knowledge about technology, design,  and manufacturing.
With the growing fiscal pressures now facing the nation, DOD needs to get  the best value for every dollar it invests in weapon system programs. The  department’s management of its weapon system programs has been a  matter of congressional concern for many years. In 2008, the  Subcommittee requested that we study how DOD makes trade offs in  requirements, costs, and technical risks before approving programs to  start development. Specifically, the Subcommittee asked us to review the  department’s Analysis of Alternatives (AOA) process—a key first step in  the acquisition process intended to assess the operational effectiveness,  costs, and risks of alternative weapon system solutions for addressing a  validated warfighting need. Determining what type of weapon system to  pursue is critical because, according to a recent estimate, about three- quarters of a program’s total life-cycle cost is influenced by decisions  made before it is approved to start development. This report assesses  (1) whether AOAs have been effective in identifying the most promising  options and providing a sound rationale for weapon program initiation,  (2) the factors that affect the scope and quality of AOAs, and (3) whether  recent DOD policy changes will enhance the effectiveness of AOAs.
To assess DOD’s AOA process, we reviewed relevant DOD and military  service policy and guidance, and interviewed officials from the Joint Staff,  Office of Program Analysis & Evaluation, Air Force Office of Aerospace  Studies, Army Training and Doctrine Command Analysis Center, and other  subject matter experts. We also collected and analyzed information on  AOAs from 32 major defense acquisition programs that had started since  fiscal year 2003. We obtained information from program officials on how  these AOAs were conducted and whether the AOAs contributed to  changes in the program’s weapon system concept. Furthermore, we  reviewed AOA documents and DOD and service guidance for conducting  these AOAs. In reviewing AOAs, we examined the scope of alternatives  that were considered and the extent of risk assessments conducted for  each alternative. This work was conducted from June 2008 to September  2009 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings   and conclusions based on our audit objectives. Additional information  about our scope and methodology is provided in appendix I; appendix III  contains information about the programs and AOAs we reviewed.
subsections: section_title: Background
paragraphs: Before service or joint sponsors can initiate major defense acquisition  programs and begin system development at Milestone (MS) B, they are  required by DOD’s acquisition policy to conduct an AOA. The AOA is an  analytical study that is intended to compare the operational effectiveness,  cost, and risks of a number of alternative potential solutions to address  valid needs and shortfalls in operational capability. The basis for  conducting an AOA begins when a capability need is validated or approved  through the department’s requirements determination process—the Joint  Capabilities Integration and Development System (JCIDS) (See fig. 1).
A sponsor, usually a military service, submits a capability proposal—called  an Initial Capabilities Document (ICD)—through JCIDS, which identifies  the existence of a capability gap, the operational risks associated with the  gap, and a recommended solution or preferred set of solutions for filling  the gap. When a capability proposal is validated, before a major defense  acquisition program begins, an AOA is undertaken to compare potential  solutions and determine the most promising and cost-effective weapon  system to acquire. The AOA is a key input to defining the system  capabilities of the major defense acquisition program, which are  established in a capability development document (CDD).
Most AOAs are sponsored by a single military service, but some may be  conducted jointly by more than one service, in which case, the Milestone  Decision Authority (MDA) designates a lead service as the sponsor. AOAs  are conducted by study teams, the composition of which depends on the  service—most of the Army’s AOAs are conducted by the Army’s Training  and Doctrine Command Analysis Center, most of the Air Force’s AOAs are  conducted by the Air Force’s major commands, such as the Air Combat  Command, and most of the Navy’s AOAs are contracted out to federally  funded research and development centers and the Navy’s various study  centers. Both the Office of the Secretary of Defense (OSD) and the  services are responsible for issuing study guidance to scope the AOA,  which provide a minimum set of alternatives to analyze and shape the  analysis through a series of study questions. Conducting an AOA may take  anywhere from a few months to several years and cost from a few hundred  thousand to several million dollars depending on its scope and complexity.  The final results and recommendations of the AOA are then presented to  decision makers, who decide on which alternative to select for program  initiation. According to the Air Force’s manual on conducting AOAs, some  of the key questions that decision makers need the AOA to answer include:    What alternatives provide validated capabilities?    Are the alternatives operationally suitable and effective?    Can the alternatives be supported?    What are the risks (technical, operational, programmatic) for each  alternative?
What are the life-cycle costs for each alternative?    How do the alternatives compare to one another?
The Office of the Secretary of Defense, Program Analysis and Evaluation  (OSD PA&E), plays a central role in the AOA process because it is  responsible for providing initial guidance to the AOA study team,  reviewing the proposed AOA study plan, and assessing the completed  AOA. In carrying out these functions, OSD PA&E provides a DOD  enterprise-level perspective to AOAs and encourages service sponsors to  consider all viable concepts to fill a capability need, even if they were not  initially considered by the service sponsors, and to assess technical risks  and costs of each alternative.
The AOA is one of several inputs required for a program’s initiation at MS  B, and it is a key element in planning and establishing a sound business  case for a weapon system program. We have frequently reported on the  importance of using a solid, executable business case before committing  resources to a new product development effort. The business case in its  simplest form is demonstrated evidence that (1) the warfighter’s needs are  valid and that they can best be met with the chosen concept, and (2) the  chosen concept can be developed and produced within existing  resources—that is, proven technologies, design knowledge, adequate  funding, and adequate time to deliver the product when it is needed. The  AOA addresses the first point of a business case by providing a foundation  for developing and refining the operational requirements for a weapon  system program. An AOA also addresses the second point of a business  case by providing insight into the technical feasibility and costs of  alternatives. By contributing to business cases, AOAs should provide  programs with a sound basis for program initiation.
section_title: Most Programs Have Not Conducted a Robust Assessment of Alternatives
paragraphs: Most of the programs we reviewed either did not conduct an AOA or  conducted an AOA that focused on a narrow scope of alternatives and did  not adequately assess and compare technical and other risks of each  alternative. While many factors can affect program cost and schedule  outcomes, we found that programs that conducted a limited assessment of  alternatives before the start of system development tended to experience  poorer outcomes than the programs that conducted more robust AOAs.  According to several DOD and program officials, AOAs have often simply  validated a concept selected by the sponsor and are not used as intended  to make trade offs among performance, cost, and risks to achieve an  optimal weapon system concept that satisfies the warfighter’s needs  within available resource constraints.
subsections: section_title: Most Programs Analyzed a Narrow Scope of Alternatives before They Started
paragraphs: Most of the programs we reviewed considered a narrow scope of  alternatives to support program start. Ten of the 32 programs did not  conduct AOAs and focused on an already selected weapon system  solution. Of the 22 programs that had AOAs, 13 of them examined a limited  number of alternatives within a single weapon system concept such as  helicopters or specific classes of ships, while 9 considered a relatively  broad range of alternatives, by assessing many alternatives within a single  weapon concept or alternatives across multiple concepts, such as  comparing ships to aircraft. We found that the programs that considered a  broad range of alternatives tended to have better cost and schedule  outcomes than the programs that looked at a narrow scope of alternatives  (see table 1). For example, 1 of the 9 programs that examined a broad set  of alternatives experienced high cost or schedule growth whereas 8 of the  13 programs that considered only a limited number of alternatives  experienced high cost or schedule growth.
For various reasons, 10 of the 32 weapon programs we reviewed did not  have formal AOAs to support program start (see table 2). For 7 of these  programs, it may have been appropriate not to conduct the AOA because  the programs involved a planned modernization to an existing weapon  system or there was support from other analyses to warrant the chosen  concept. This was the case, for example, with the Navy’s Standard Missile  6 (SM-6) program. Because the missile was the next planned increment in  a long history of missile development efforts and an AOA had been  conducted for the previous standard missile increment, a separate AOA for  SM-6 was considered repetitive and waived. The program started  development in 2004 and has remained on track with its planned cost and  schedule objectives. Similarly, an AOA was not conducted for the Air  Force’s Global Positioning System IIIA program because there was a body  of analysis available that served the purpose of an AOA and the proposed  program was considered a follow-on increment to a multiprogram effort to  modernize global positioning system capabilities. Since it started  development in 2008, the program has remained on cost and schedule.
However, in the other 3 programs that did not have AOAs, the  requirements and development effort proved to be more demanding and  cost and schedule growth occurred. In the case of the Army’s Sky Warrior  Unmanned Aerial System (UAS) program, an Army executive waived the  AOA requirement because the Army believed, among other things, that the  source selection process would provide an adequate way to compare  alternatives. However, when the Air Force and Joint Staff were reviewing  the Sky Warrior’s draft requirements and acquisition documentation, they  raised concerns that the requirements potentially duplicated capability  provided by the Air Force’s Predator UAS. The Army cited the urgent need  of battlefield commanders for the capability and gained approval to  proceed to source selection. Three years after the Sky Warrior AOA was  waived, the Deputy Secretary of Defense directed that the two UAS  programs be combined into a single acquisition program to achieve  efficiencies in areas such as common development, procurement, and  training activities. However, the Army and Air Force have continued to  pursue unique systems. In the meantime, the Sky Warrior UAS has  experienced a 138 percent increase in total cost and 47-month schedule  delay from original plans. By relying on industry-provided information in  source selection and not conducting an independent AOA, the Army  missed an opportunity to gain a better understanding of the other services’  UAS capabilities, and pursue an acquisition strategy that would have taken  advantage of commonalities and used resources more efficiently.
subsections: section_title: Programs That Conducted AOAs
paragraphs: Of the 22 programs that conducted AOAs, 13 focused on a limited number  of alternatives within a single weapon system concept while 9 focused on  many alternatives (see table 3). According to DOD and service officials,  the scope of an AOA can be different for each program and dependent  upon many factors, including the nature of the capability need, the  proposed time frame for fielding the capability, and the type of program  being pursued–whether it is a new development start, a modification of a  commercially available system, or an upgrade to an existing system. As a  result, AOAs that focus on a limited number of alternatives within a single  weapon system concept may be appropriate in some cases. For instance,  when the capability need was defined in terms of upgrading an existing  weapon system, AOAs focused on refining a single platform concept and  its system-level specifications and attributes. The AOA for the Army’s  Apache Block III program is an example of an appropriately, but narrowly  scoped AOA. It examined various block upgrade options for the existing  Longbow Apache helicopter to improve interoperability and other  shortcomings in the helicopter. The program started development in 2006  and has remained on track with its planned cost and schedule objectives.
In a few of the other AOAs that had a narrow scope, the capability need  involved the replacement of an aging weapon system and the AOAs  presumed that the concept of the aging weapon system was the  appropriate starting point for analysis rather than examining whether  other concepts could also meet the need. For example, the AOA for the  Army’s Armed Reconnaissance Helicopter (ARH) program, which was  intended to replace the aging Kiowa helicopter fleet and improve attack  and reconnaissance capabilities, examined two options: improving the  legacy Kiowa helicopter or procuring nondevelopmental helicopters. The  AOA did not explore other potential solutions, such as developing  unmanned aerial systems, increasing the purchase of existing attack  helicopters, increasing the purchase of other reconnaissance assets, or  relying on a mix of solutions. After 3 years of development, the ARH  program’s research and development costs increased from about $360  million to $940 million. A Center for Naval Analyses report commissioned  by the Army after the ARH program began having execution problems  identified several factors that contributed to the significant cost growth,  including questionable requirements, an aggressive schedule, limited  oversight, and a perceived preference for one helicopter model. As a result  of the cost growth and other problems, DOD cancelled the program in  2008 after determining that at least one alternative could provide equal or  greater capability at less cost.
Most of the programs (7 of 9) that examined a broad scope of alternatives  have tracked well with their planned cost and schedule targets. The AOA  for the Navy’s P-8A Multi-mission Maritime Aircraft, which is a program  designed to replace the P-3C aircraft and provide maritime patrol and  reconnaissance for the Navy, explored multiple concepts and many  alternatives in response to study guidance issued by OSD PA&E, including  several nonmanned aircraft alternatives such as submarines, helicopters,  and UAS. The AOA concluded that a manned aircraft would still be the  best option to replace the P-3C. However, the AOA also helped the Navy to  recognize that a UAS could perform some of the maritime patrol missions  as an adjunct platform, eventually leading to the Broad Area Maritime  Surveillance (BAMS) UAS AOA and program. The P-8A program has not  experienced cost growth over its 4 years of development and remains on  schedule. Similarly, the Joint Land Attack Cruise Missile Defense Elevated  Netted Sensor System (JLENS), which is designed to provide over the  horizon detecting and tracking of land attack cruise missile and other  targets, had an AOA that explored alternatives across multiple concepts,  including aerostat sensors, sea-based sensors, and nonaerostat elevated  sensors. The Army chose the aerostat concept and has developed an  incremental program that has experienced low cost and schedule growth  since starting development in 2005.
section_title: Many AOAs Have Not Adequately Assessed Risks for the Alternatives
paragraphs: DOD acquisition policy requires that AOAs assess the technical risk of  alternatives, but it does not provide criteria and guidance for how and to  what extent technical risks should be addressed and it does not specify  that other types of risks should be assessed. Risks are important to  assess because there may be technical, programmatic, or operational  uncertainties associated with different alternatives that should be  considered in determining the best weapon system approach. For  example, it may be the case that one alternative is more effective than  another in meeting a capability need but has more technical or other risks  that may make the alternative infeasible to develop. Many of the AOAs we  reviewed (12 of the 22) conducted limited assessments of the risks of each  alternative presented (see table 4). Some AOAs we reviewed did not  examine risks at all, focusing only on the operational effectiveness and  costs of alternatives. Other AOAs had relatively limited risk assessments.  For example, several AOAs did not discuss integration risks even though  they were examining modified commercial systems that required the  integration of subsystems or equipment packages, while other AOAs did  not examine the schedule risks of the various alternatives, despite  accelerated schedules and fielding dates for the programs. We found that  programs with AOAs that conducted a more comprehensive assessment of  risks tended to have better cost and schedule outcomes than those that did  not (see table 5).
AOAs that do not examine risks could provide overly optimistic  assessments of alternatives, which do not provide for sound business case  decisions. Comparing risks across alternatives is especially critical for new  development programs, which rely on breakthrough technologies and  assume that technology will be achieved as planned. Of the 22 programs  that had AOAs, 8 were new development starts involving technology  development. Of the 8 new development starts, only 4 had AOAs that  performed adequate risk analyses. The other 4 AOAs did not assess  technical, integration, or other risks as criteria for comparing the  alternatives or neglected to analyze these risks altogether. For example,  the AOA for the Future Combat Systems (FCS), one of most complex and  technically challenging programs ever undertaken according to the Army,  assessed the technical risks of each of the new development concepts for  FCS, but did not assess and compare the risks with those of the other  alternatives. The AOA concluded that the new FCS development option  was more costly but more operationally effective than the baseline and  improved baseline alternatives. By not comparing the risks of the  alternatives, the FCS AOA missed an opportunity to provide the Army with  a meaningful trade off among operational effectiveness, costs, and risks.  Now, after 6 years of development, some of the critical technologies for  the FCS program are still immature. The latest estimates for the program  show that development costs have grown 38 percent or about $8 billion,  and the fielding date has been delayed 57 months. As a result, DOD  recently proposed canceling the FCS acquisition program.
Also, the AOA for the Army’s Warfighter Information Network-Tactical  (WIN-T) program, which involves development of new on-the-move  networking capabilities, did not address technical or programmatic risks.  Army officials stated that WIN-T was largely based on a concept that did  not have well-defined requirements of the proposed network and  operations, and the WIN-T development alternative in the AOA was based  on preliminary design concepts, from two competing contractors, which  were blended together by the Army. The AOA did not take these risks into  account and concluded that the new WIN-T alternative was the most  operational and cost-effective solution available. In March of 2007, the  WIN-T program had a Nunn McCurdy cost breach (25 percent or more unit  cost growth) and was subsequently restructured by DOD. Insufficient  technical readiness was cited as one of the key factors leading to the cost  breach.
Assessing risks is also important for programs based on commercial  products that require significant modifications. Based upon a recent  Defense Science Board report on buying commercially-based defense  systems, programs that do not assess the systems engineering and  programmatic risks of alternatives do not understand the true costs  associated with militarizing commercial platforms or integrating various  commercial components. As a result of this incomplete understanding of  inherent technical and integration risks of programs, DOD fails to fully  take advantage of efficiencies and cost savings from commercially  available technologies. Several of the programs we reviewed that involved  modified commercial products had AOAs with weak risk assessments. For  example, the AOA for the Marine Corps’ replacement for the Presidential  Helicopter, VH-71, failed to assess the technical, integration, and schedule  risks associated with its three alternatives. It instead compared  alternatives based on costs and performance attributes, such as cabin size,  deployability, and performance. One program official stated that the focus  of the VH-71 AOA was to merely identify platforms that had the best  probability of meeting the requirements. According to a statement by the  Secretary of Defense, the program’s costs have nearly doubled, increasing  from $6.5 billion to $13 billion, and the schedule has fallen behind by  several years. DOD recently cancelled the program. The Defense Science  Board, which assessed the VH-71 program, concluded that some of the  program’s requirements plainly exceeded the limits of the available  technology and schedule.
section_title: Choosing an Alternative Too Early and Conducting AOAs under Compressed Time Frames and without Effective Guidance Limit the Scope and Quality of AOAs
paragraphs: We identified several factors that may have limited the effectiveness of  AOAs and their ability to identify the most promising option and  contribute to a sound business case for starting a weapon system program:  (1) service sponsors lock into a solution early on when a capability need is  first validated through DOD’s requirements process and before an AOA is  conducted; (2) AOAs are conducted under compressed time frames in  order to meet a planned milestone review or fielding date and their results  come too late to inform key trade off decisions; and (3) DOD does not  always provide guidance for conducting individual AOAs. The AOAs with  one or more of these factors tended to be AOAs that had a limited scope  and assessment of risks (see table 6).
subsections: section_title: Service Sponsors Lock Into a Program Solution or Establish Requirements Prior to the AOA
paragraphs: In developing a capability proposal, sponsors not only justify the need to  fill an existing capability gap, but also conduct an assessment—called a  functional solutions analysis (FSA)—to identify a potential concept or set  of solutions to fill the gap. The identification of a potential concept is  intended to provide a general approach for addressing the gap and set the  stage for a more in-depth assessment of alternatives to be conducted in the  AOA. In four cases, AOAs were limited because program sponsors had  decided on a preferred solution prior to the AOA, when a capability need  was first proposed through the department’s requirements determination  process. Approval of the capability proposal then led to a narrowly scoped  AOA that supported or refined the preferred solution. According to DOD  officials, the analysis supporting a capability proposal is generally  conducted by the operational requirements community within a military  service and contains only rudimentary assessments of the costs and  technical feasibility of the solutions identified.
With the Armed Reconnaissance Helicopter program, for example, the  Army proposed acquiring an armed reconnaissance helicopter after the  termination of the Comanche helicopter program, which had experienced  significant cost and schedule problems. While the initial capability  proposal submitted to JCIDS for the ARH considered nonhelicopter  concepts, such as unmanned aerial systems, the Army concluded that a  modified version of an existing armed reconnaissance helicopter was the  preferred solution. According to Army officials, the modified helicopter  solution was pushed in part because there was a desire to field a system  within a relatively short time frame, a similar helicopter variant was in use  by the special operations forces, and funding available from the  terminated Comanche helicopter program needed to be used quickly.  Because the Army effectively locked into a solution in this early stage, the  AOA primarily focused on comparing the performance and costs of  existing helicopter alternatives (see fig. 2).
Armed Reconnaissance Helicopter (ARH)
Similarly, we have previously reported that the Navy began the Littoral  Combat Ship (LCS) program before fully examining alternatives.  Beginning in 1998, the Navy conducted a series of wargames and studies to  test new concepts for surface combatant ships that could address known  threats in littoral areas. Following these efforts, the Navy began an  analysis of multiple concepts study in 2002 to further refine the Navy’s  preferred solution—a new warship along the lines of LCS. Concurrently,  the Navy established an LCS program office and issued a request for  proposal to industry to submit LCS concepts. The Office of the Secretary  of Defense and the Joint Staff were concerned that the Navy’s focus on a  single solution did not adequately consider other ways to address littoral  capability gaps. Based on these concerns in late 2003, the Navy was  directed to consider alternatives to surface ships such as submarines and  manned aircrafts in the ongoing analysis of multiple concepts. The  analysis, which was led by the Naval Surface Warfare Center, compared  nonship alternatives to LCS-concept ships and concluded that the LCS  concept remained the best solution to provide capabilities in the littorals.  However, the estimated costs for the various LCS ship alternatives  developed in the analysis far exceeded the $220 million (fiscal year 2005  dollars) target that the Navy had set for the program. The Navy stated that  because the cost estimates were rough-order-of-magnitude estimates and  were based on preliminary concept designs, those costs were not used to  make cost decisions for LCS. However, since starting development in 2004,  the LCS program has experienced a 151 percent growth in development  costs and its costs are closer to the cost estimates from the analysis of  multiple concepts than the target cost set by the Navy.
DOD and service officials responsible for conducting AOAs indicated that  often capability requirements are proposed that are so specific that they  effectively eliminate all but the service sponsor’s preferred concepts  instead of considering other alternatives. For example, in recent proposals  to address a global strike capability need, two components of the Air  Force—the Air Combat Command and Space Command—defined initial  performance requirements that required two different approaches. The Air  Force Air Combat Command defined the requirement as the ability to  strike a target within 1 day, which meant that bombers, which fall under  the Air Force Air Combat Command’s portfolio, could address the gap.  However, the Air Force Space Command defined the requirement in the  capability proposal as the ability to strike a target within a certain number  of hours, which meant only missiles, which fall under the Air Force Space  Command, could fulfill the need. Although OSD PA&E attempted to get  the Air Force to consider both bombers and missiles in the same analysis,  the major commands argued that their requirements were different enough  to require two separate analyses. As a result, the Air Force Air Combat  Command initiated the Next Generation Long-Range Strike AOA for a new  bomber, while the Air Force Space Command initiated the Prompt Global  Strike AOA separately.
Similarly, for the ARH AOA, the Army called for very specific deployability  requirements. These requirements included the ability to fit two  helicopters into a C-130 aircraft and for the helicopter to be “fightable”  within 15 minutes of arrival. The Center for Naval Analyses, in its report  on the factors that led to significant cost and schedule growth in the ARH  program, noted that it was not clear whether these requirements were  needed to fulfill the operational gap. Furthermore, the Center for Naval  Analyses noted that due to the stringent deployability requirements, the  program had effectively eliminated other potentially feasible and cost- effective alternatives, such as twin-engine helicopters, and limited the  analysis to single engine alternatives.
section_title: Timing of AOAs May Not Be Conducive to Informing Trade Offs
paragraphs: Many AOAs are also conducted under compressed time frames—6 months  or less—or concurrently with other key activities that are required for  program initiation, in order to meet a planned milestone decision or  weapon system fielding date. Consequently, AOAs may not have enough  time to assess a broad range of alternatives and their risks, or be  completed too late in the process to inform effective trade discussions  prior to beginning development. In 9 of the 22 programs we reviewed that  had AOAs, the timing of the AOAs was compressed or concurrent with  other planning activities. In 7 of these 9 programs, the AOAs were limited.  For instance, the AOA for the Future Combat Systems program was a  complex undertaking; however, according to the authors of the AOA, it  was conducted in half the time that a less complex AOA would typically be  conducted. In addition, due to schedule constraints imposed to meet a  preset milestone review date, the AOA was performed concurrently with  concept development, requirements determination, and system definition  documents. Ultimately, the Future Combat Systems AOA was completed 1  month after the operational requirements were validated and the same  month that the program was approved to begin system development,  which precluded trade off discussions among cost, performance, and risks  from taking place. In addition, although AOAs are required to be done for a  Milestone B decision, the Army’s Warfighter Information Network-Tactical  (WIN-T) program was approved to begin without one. The milestone  decision authority for the program waived the AOA requirement until a  later date. The WIN-T AOA was completed approximately 16 months after  the program started (see fig. 3).
section_title: DOD Has Not Consistently Provided Guidance for Conducting Individual AOAs
paragraphs: While DOD acquisition policy requires that major defense acquisition  programs conduct an AOA prior to program initiation at Milestone B, the  policy does not specify criteria or guidance for how AOAs should be  conducted. According to the policy, OSD PA&E is to provide guidance to  programs prior to, during, and after their AOA has been completed. The  guidance is intended to ensure that the services are examining a sufficient  number of alternatives that take into consideration joint plans and  interoperability, but to also ensure that AOAs are analyzing key risks such  as technology, cost, and schedule. In 9 of the 22 programs we reviewed  that had AOAs, OSD PA&E either provided late guidance or did not  provide formal guidance when AOAs were started. In 6 of these 9  programs, the AOAs were limited. For instance, OSD PA&E did not  provide guidance for the AOA that supported initiation of the VH-71  Presidential Helicopter program. In this AOA, the service had very specific  performance requirements that narrowed the scope of the alternatives  examined. In addition, the service conducted the AOA under a compressed  schedule to meet a previously planned milestone, which may not have  allowed for robust analyses of technology and integration risks. These  factors most likely played a part in the AOA examining only 3 alternatives  and eliminating 19 other alternatives early on.
DOD officials have also stated that when OSD PA&E guidance is provided,  it is sometimes late. For example, the LCS program AOA had been  underway for about a year before OSD PA&E provided guidance to the  Navy. Officials also explained that guidance is often informal, sometimes  provided over the telephone, or if written, remains in draft form for long  periods, preventing the services from formulating and having analysis  plans approved. However, according to PA&E officials, sometimes  guidance is never formalized or written because the services do not have a  validated capability proposal or do not agree with the scope and direction  provided. By not providing timely formal guidance before AOAs are  started, DOD is missing an opportunity to ensure AOAs examine an  appropriate scope of alternatives and conduct robust risk assessments.
section_title: Recent DOD Policy Changes Could Improve AOA Effectiveness
paragraphs: In December 2008, DOD revised its acquisition policy and introduced  several initiatives based in part on direction from Congress that could  provide a better foundation for establishing knowledge-based business  cases for initiating weapon system programs. The revised policy  strengthens the front end of the acquisition process by requiring key  systems engineering activities and early prototyping, and establishing  required milestone reviews to assess whether programs are acquiring the  requisite knowledge as they move towards the start of system  development. In addition, in March 2009, DOD revised its policy governing  the JCIDS process, to help streamline the determination of capability  needs and improve the integration between JCIDS and the acquisition  process. In revising these policies, DOD elevated the role of AOAs in  determining weapon system concepts and strengthened how they are to be  implemented. Improving the effectiveness of AOAs will depend on DOD’s  ability to ensure that its policy changes are consistently implemented and  reflected in decisions on individual weapon system programs. We have  reported in the past that inconsistent implementation of existing policies  has hindered DOD’s efforts to plan and execute programs effectively. The  key revisions to the policies that impact AOAs are summarized in table 7.
DOD’s revised policies, for example, may help mitigate service sponsors  from locking into a solution too early in the process by eliminating the  functional solutions analysis in a capability proposal, which identified a  preferred solution and influenced the scope of alternatives in an AOA. In  the revision, the capability proposal will only identify a broad category of  the type of materiel solution that should be considered; for example,  whether it should be an incremental or transformational development  approach. The AOA will then assess potential solutions as determined by  the milestone decision authority and within the broad category  recommended. This change integrates essentially what had been two  separate trade space analyses into one analysis. In doing so, it sets up a  better opportunity for a more robust analysis of alternatives.
DOD’s revised acquisition policy also now imposes early milestone  reviews which should help resolve the timing issues we found with several  AOAs in the past. Under the previous policy, AOAs were required for  program initiation at Milestone B, which may have led to some AOAs being  completed just prior or even after program initiation. Under the revised  policy, AOAs are generally required earlier in the process. Furthermore,  DOD PA&E is required to be involved much earlier in the process by  providing requisite guidance at the Materiel Development Decision as well  as approving AOA study plans before an AOA is started. These additional  reviews with required guidance earlier in the acquisition process should  help mitigate conducting AOAs under compressed time frames. However,  while the revised policy strengthens the front end of the acquisition  process, the AOA is still constrained to a given set of requirements that  may be unfeasible and could lead to unsuccessful program outcomes, such  as with the Armed Reconnaissance Helicopter and Future Combat  Systems.
section_title: Conclusions
paragraphs: With increased demand and competition for funding, it is critical that DOD  weapon system programs provide the best value to the warfighter and to  the taxpayer. Yet in too many cases, DOD programs do not accomplish this  and experience significant cost, schedule, and performance problems.  Many of these problems could be avoided if programs started with sound,  knowledge-based business cases. A key to developing such business cases  is having effective AOAs that analyze and compare the performance, costs,  and risks of competing solutions, and identify the most promising weapon  system solution to acquire. The majority of AOAs we reviewed were  limited and thus did not sufficiently inform the business case for starting  new programs.
DOD’s recent policy revisions are positive steps that could, if implemented  properly, provide a better foundation for conducting AOAs and  establishing sound business cases for starting acquisition programs. The  revisions, for example, should help ensure that DOD direction is provided  before AOAs are started and that AOAs are conducted at an early point in  the acquisition process where their results can inform key decisions  affecting program initiation. However, these policy changes alone will not  be sufficient to ensure AOAs achieve their intended objectives. Unless  mechanisms are established to ensure policy is followed, specific guidance  and criteria are developed for how AOAs should be conducted, and AOAs  are completed before program requirements are set, AOAs will not provide  effective in-depth analyses and DOD will continue to struggle to make  informed trade offs and start executable programs.
section_title: Recommendations for Executive Action
paragraphs: To further strengthen the effectiveness of AOAs in helping DOD establish  sound business cases for major weapon programs, we recommend that the  Secretary of Defense take the two following actions:    Establish specific criteria and guidance for how AOAs should be  conducted, including how technical and other programmatic risks should  be assessed and compared.
Ensure that AOAs are completed and approved before program  requirements—key performance parameters and attributes—are finalized  and approved.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD concurred with our  recommendations. DOD’s response is reprinted in appendix II. DOD stated  in response to our first recommendation that it had made significant  progress in establishing criteria and guidance for conducting AOAs, and in  defining the relationship/role of the AOA in both the acquisition and  capabilities determination processes. DOD indicated that the role of the  AOA has been defined in recently revised acquisition policy (Department  of Defense Instruction 5000.02, dated Dec. 2, 2008) and capabilities policy  (Chairman, Joint Chiefs of Staff Instruction 3170.01G, dated Mar. 1, 2009).  While we agree that promising improvements have been made in revising  the policies, they do not go far enough in providing specific criteria and  guidance for how AOAs should be conducted. Without such direction,  there is a risk that AOAs will continue to provide limited assessments of  weapon system options, and DOD will initiate programs without sound,  executable business cases. In concurring with our second  recommendation—that AOAs be completed before requirements are  finalized—DOD pointed out that under its revised acquisition policy, AOAs  are now required to be completed before the formal initiation of an  acquisition program. We agree that the policy should help improve the  timing of AOAs so that they are conducted at an early point in the  acquisition process and provide an opportunity for trade offs to take place.  However, establishing and approving requirements is another key step  required for initiating an acquisition program and this is done under a  separate process—the Joint Capabilities Integration and Development  System. We believe that DOD needs to take steps to ensure that program  requirements are not finalized before the AOA is completed and that the  results of the AOA are used to inform the setting of requirements.
DOD also provided technical comments, which we incorporated where  appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the Secretary  of Defense; the Secretaries of the Air Force, Army, and Navy; and  interested congressional committees. This report will also be available at  no charge on the GAO Web site at http://www.gao.gov.
If you have any questions about this report or need additional information,  please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made major contributions to  this report are listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To assess whether analyses of alternatives (AOA) have been effective in  identifying the most promising options and providing a sound rationale for  program initiation, we analyzed data and documents for Acquisition  Category (ACAT) I programs that have been initiated between fiscal years  (FY) 2003 and 2008 and were in the Department of Defense’s (DOD) FY  2008 Major Defense Acquisition Program (MDAP) list. The relevant policy  that governs the AOA process for these programs, DOD Instruction 5000.2  (Operation of the Defense Acquisition System), was revised by DOD in  May of 2003 and revised again in December of 2008 to become DOD  Instruction 5000.02. As a result, we used the May 2003 DOD Instruction to  assess the AOAs. Using DOD’s FY 2008 MDAP list and Milestone B dates  provided by DOD, we identified 34 ACAT I programs that had been  initiated, or started system development and production, between 2003  and 2008. Programs that had been initiated between 2003 and 2008 but  were not in the FY 2008 MDAP list, such as programs terminated before  2008, were not included in the analysis. We collected AOA full reports,  executive summaries, guidance documents, and study plans when  available, from program officials. Program officials also responded to data  collection surveys we distributed through service action officers to gather  information about their programs’ AOA, guidance, capability documents,  and how the AOA led to changes to the program concept. An official for  the Cobra Judy Replacement program responded to the survey, but  officials did not respond to several phone calls and e-mails requesting  additional documentation, so this program was not included in the  analysis. In addition, because the Combat Search and Rescue Replacement  Vehicle (CSAR-X) program did not start development.
Of the remaining 32 programs, 10 programs did not have AOAs. Whether a  program had an AOA or not was determined through analysis of program  documents and survey responses. For the 22 programs that had AOAs,  program documents and survey data were reviewed to determine the  scope of the AOAs and whether the AOA assessed technology and  integration risks. An AOA’s scope was assessed to be narrow if the AOA  examined 2 to 5 alternatives within a single concept and assessed to be  broad if the AOA examined 8 to 26 alternatives within a single concept or  multiple concepts. An AOA was assessed to have not completed any risk  analyses for its alternatives when it made no mention of risks in the entire  AOA report; assessed to be limited if the risk analyses were not completed  for all of the alternatives, if integration risks were not examined, or if the  risk analyses were not emphasized in the conclusions and  recommendations; and assessed to be adequate if technical and integration  risks were analyzed and compared for all of the alternatives. We followed  up with some program officials through phone calls and e-mails for  additional information. To assess how the quality of AOAs correlates with  programs’ outcomes, we also collected program and cost data from DOD’s  Selected Acquisition Reports and GAO’s Annual Assessments of Selected  Weapon Programs. Programs with less than 10 percent cost growth were  considered to have low cost growth, programs with 10 to 24 percent cost  growth were considered to have moderate cost growth, and programs with  25 percent or more cost growth were considered to have high cost growth.  Programs with less than 7 months of delay in initial operational capability  or acquisition cycles were considered to have low schedule growth,  programs with 7 to12 months of delay in initial operational capability or  acquisition cycles were considered to have moderate schedule growth,  and programs with greater than 12 months of delay in initial operational  capability or acquisition cycles were considered to have high schedule  growth. The 32 programs we reviewed accounted for one third of the 96  programs in DOD’s 2008 Major Defense Acquisition Program portfolio and  approximately 22 percent of the total planned funding commitments.
To identify the factors that have affected the scope and quality of AOAs,  we reviewed program documents, analyzed data from the survey, and  reviewed DOD policy. We reviewed Initial Capabilities Documents (ICD)  gathered from the Joint Staff’s Knowledge Management/Decision Support  tool and AOAs to determine how preferred solutions were carried from the  requirements-generation process to the acquisition process. To determine  how program schedules affected AOA scope and methodology, we  analyzed AOA documents, program milestone dates, and AOA completion  dates. To assess how DOD study guidance affected the quality of AOAs, we  analyzed whether DOD provided guidance through survey responses and  followed up with DOD to confirm those responses. We also reviewed  regulations and policies issued by the Joint Staff, the military services, and  DOD, as well as other DOD-produced documentation related to AOAs.
To determine what additional actions may be needed to address the  limitations in the AOA process, we analyzed relevant DOD policies and  federal statutes, including DOD Instruction 5000.2 (May 2003), DOD  Instruction 5000.02 (December 2008), the Chairman of the Joint Chiefs of  Staff Manual (CJCSM) 3170.01C (May 2007), CJCSM 3170.01 (March 2009),  and Section 2366a of Title 10, United States Code.
In researching all three objectives, we interviewed officials from the U.S.  Army G3; U.S. Army Training and Doctrine Command Analysis Center  (TRAC); U.S. Army Capabilities Integration Center (ARCIC); U.S. Air Force  Office of Aerospace Studies; Office of the Assistant Secretary for  Acquisition, Deputy Assistant Secretary of the Air Force for Science,  Technology, and Engineering; Air Force Acquisitions - Global Reach;  Deputy Assistant Secretary of the Navy, Acquisition and Logistics  Management (A&LM); Deputy Directorate for Antiterrorism and Homeland  Defense, J-34, Joint Staff; Office of the Secretary of Defense, Acquisition,  Technology & Logistics; Office of the Secretary of Defense, Program  Analysis and Evaluation; Office of the Deputy Under Secretary of Defense  for Science and Technology (Acquisition and Technology)/Systems and  Software Engineering; Armed Reconnaissance Helicopter Product  Manager’s Office; U.S. Army Aviation Center; Deputy Assistant Secretary  of the Navy, Ship Programs; Littoral Combat Ship Program Office; Marine  Corps Combat Development Command; Office of the Chief of Naval  Operations, Deputy Chief of Naval Operations, Integration of Capabilities  and Resources (N8), Director of Warfare Integration (N8F), Director of  Surface Warfare (N86); Air Combat Command/A8I (Requirements),  Secretary of the Air Force Technical and Analytical Support; and the Naval  Surface Warfare Center, Dahlgren.
We conducted this performance audit from June 2008 to September 2009  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: Key Characteristics of AOAs
paragraphs: We surveyed 32 major defense acquisition programs on their analyses of  alternatives process and outputs. Ten of the programs did not conduct  AOAs. The following table provides characteristics of the 22 programs that  conducted AOAs.
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, the following individuals made  key contributions to this report: John Oppenheim (Assistant Director),  Martin G. Campbell, James Kim, John Krump, Claire Li, Guisseli Reyes- Turnell, and Tatiana Winger.