Weapons Acquisition Reform: Reform Act Is Helping DOD Acquisition Programs Reduce Risk, but Implementation Challenges Remain
Published: 2012-12-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: For the past 3 years, DOD has been implementing the Reform Act requirements which are aimed at helping weapon acquisition programs establish a solid foundation from the start. This helps to prevent cost growth, thus helping the Defense dollar go further. This is the third in a series of GAO reports on the Reform Act.
GAO was asked to determine (1) DOD's progress in implementing Reform Act provisions; (2) the impact the Reform Act has had on specific acquisition programs; and (3) challenges remaining. To do this, GAO analyzed documents and interviewed officials from the four OSD offices created as a result of the Reform Act, other DOD offices, the military services, and 11 weapon acquisition programs we chose as case studies. Case study programs were selected based on their development status and interaction with the four OSD offices. Results cannot be generalized to all DOD weapon acquisition programs.
section_title: What GAO Found
paragraphs: The Department of Defense (DOD) has taken steps to implement fundamental Weapon Systems Acquisition Reform Act of 2009 (Reform Act) provisions, including those for approving acquisition strategies and better monitoring weapon acquisition programs. DOD is also continuing to take additional steps to strengthen policies and capabilities. Some provisions, such as issuing guidance for estimating operating and support costs, are being implemented.
GAO's analysis of 11 weapon acquisition programs showed the Reform Act has reinforced early attention to requirements, cost and schedule estimates, testing, and reliability. For example, prior to starting development, an independent review team raised concerns about the Ground Combat Vehicle program's many requirements and the risks associated with its 7-year schedule. Subsequently, the Army reduced the number of requirements by about 25 percent and prioritized them, giving contractors more flexibility in designing solutions. In addition, the developmental test and evaluation office--resulting from the Reform Act--used test results to help the Joint Light Tactical Vehicle program develop a more realistic reliability goal and a better approach to reach it.
While DOD has taken steps to implement most of the fundamental Reform Act provisions, some key efforts to date have been primarily focused on DOD's largest weapon acquisition programs. DOD faces five challenges--organizational capability constraints, the need for additional guidance on cost estimating and Reform Act implementation, the uncertainty about the sufficiency of systems engineering and developmental testing resources, limited dissemination of lessons learned, and cultural barriers between the Office of the Secretary of Defense (OSD) and the military services--that limit its ability to broaden the Reform Act's influence to more programs. Service officials believe additional guidance is needed to improve their cost estimates and other implementation efforts. They also believe that lessons learned from programs that experience significant cost and schedule increases should be shared more broadly within the acquisition community. These challenges seem straightforward to address, but they may require more resources, which have been difficult to obtain. Ensuring the services have key leaders and staff dedicated to systems engineering and developmental testing activities, such as chief engineers at the service level and technical leads on programs, as well as breaking down cultural barriers are more difficult to address. They will require continued monitoring and attention by the Under Secretary for Acquisition, Technology and Logistics, service acquisition executives, and offices established as a result of the Reform Act to address.
section_title: What GAO Recommends
paragraphs: GAO recommends DOD develop additional cost estimating and Reform Act implementation guidance; make lessons learned available to the acquisition community; and assess the adequacy of the military services' systems engineering and developmental testing workforce. DOD generally concurred with the recommendations. GAO clarified one recommendation to make it clear that DOD needs to designate an office(s) within the Acquisition, Technology and Logistics organization to provide practical Reform Act implementation guidance to program offices.

section_title: Letter
paragraphs: For several decades, GAO has reported on poor outcomes  encompassing cost and schedule growth on the Department of Defense’s  (DOD) major weapon acquisition programs. Many problems can be traced  to a culture where the military services begin programs with inflexible  requirements, immature technologies, and overly optimistic cost and  schedule estimates. Given pressures to reduce spending across the  government, including DOD, finding ways to prevent or mitigate cost  growth is crucial to U.S. national security. A solid program foundation  using good developmental testing and systems engineering, and reliable  cost estimates is needed in order to help avoid cost overruns and  promote better acquisition outcomes. There have been numerous  attempts in the past to improve DOD acquisition outcomes, including the  Packard Commission and the Goldwater-Nichols Act in the 1980s and  Federal Acquisition Streamlining Act of 1994. More recently, Congress  passed the Weapon Systems Acquisition Reform Act of 2009 (Reform  Act) to improve the way weapon systems are acquired and avoid cost  and schedule overruns.
In 2009, the Senate Armed Services Committee asked us to begin  reporting on DOD’s implementation of Reform Act provisions and the  impact the Reform Act has had on weapon acquisition programs. This is  our third report addressing these topics. The first report focused on  DOD’s initial efforts to implement Reform Act provisions for systems  engineering and developmental testing, including the placement of new  offices for these activities within the Office of the Secretary of Defense  (OSD). Our second report examined the challenges the services face as  they try to strengthen systems engineering and developmental testing  activities on weapon acquisition programs. This report examines   (1) DOD’s progress in implementing Reform Act provisions; (2) the impact  the Reform Act has had on specific acquisition programs; and   (3) challenges remaining in improving the weapons acquisition process.
To determine DOD’s progress in implementing Reform Act provisions we  interviewed officials and analyzed documentation from the four offices  whose leadership was established within OSD as a result of the Reform  Act—Systems Engineering (SE), Developmental Test and Evaluation  (DT&E), Cost Assessment and Program Evaluation (CAPE), and  Performance Assessments and Root Cause Analyses (PARCA). We  focused our review on the offices’ implementation of four fundamental  Reform Act provisions: developing policy and guidance; approving  acquisition documents; monitoring programs and conducting program  assessments; and developing performance measures. We also analyzed  documentation we collected on selected weapon acquisitions. To  determine the Reform Act’s impact on specific defense acquisition  programs, we chose as case studies, 11 weapon systems at various  points in the development process based on recommendations by OSD  officials and GAO’s previous evaluations of these programs. At the time of  our case study selection, all of the weapon systems in our review had  been given Milestone A approval. This approval allows the programs to  start the technology development phase of the acquisition process. Four  of the 11 weapon systems had not yet received Milestone B approval.  This approval signifies the start of engineering and manufacturing  development activities. We believe these four programs offer the best  insight into how the OSD offices and Reform Act policies are influencing  acquisition strategies. The other seven programs were conducting  engineering and manufacturing development activities or had been given  Milestone C approval to begin production and deployment phase  activities. Each of these programs provided different insights in their  interactions with the OSD offices, but some Reform Act provisions, such  as conducting preliminary design reviews prior to Milestone B, may not  apply to these programs since they were beyond development start. A  complete list of programs we reviewed is provided below.
To determine challenges remaining for implementing the Reform Act, we  conducted our own assessment and interviewed the leaders of the OSD  offices established as a result of the Reform Act, military service  acquisition chiefs, and program managers. These officials are responsible  for advising and overseeing weapon acquisition program development,  funding and developing new weapon acquisition programs, and executing  the day-to-day development plans, respectively. We also solicited the  opinions of the Under Secretary of Defense for Acquisition, Technology  and Logistics and other senior OSD leaders. Additional information about  our scope and methodology can be found in appendix I.
We conducted this performance audit from January 2012 to December  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: In May 2009, Congress passed the Weapon Systems Acquisition Reform  Act of 2009 (Reform Act) in an effort to improve the way weapon systems  are acquired and avoid further cost overruns on such programs. When  signing the Reform Act into law, the President stated that its purpose was  to limit weapon system cost overruns and that it would strengthen  oversight and accountability by appointing officials who will closely  monitor the weapons systems acquisition process to ensure that costs  are controlled.
Four offices were established as a result of the Reform Act: SE, DT&E,  CAPE and PARCA. The SE and CAPE offices existed under other  organizational titles prior to the Reform Act. Staffing levels, following the  Reform Act, remained relatively stable for both of these offices, but some  reorganization was necessary to reflect new Reform Act responsibilities.  The DT&E and PARCA offices were newly established. The key roles and  responsibilities of these four offices as outlined in the Reform Act are  explained below:  Each of the offices has varying levels of interaction with defense  acquisition programs. For example, the SE and DT&E offices have  ongoing interaction with acquisition programs throughout development in  the form of program reviews, working group activities, and other program  meetings. They also coordinate program documents in preparation for  major milestone reviews. CAPE issues guidance to programs on how to  conduct an analysis of alternatives at the beginning of the acquisition  process. The office approves the analysis of alternative study plan that is  developed based on its guidance. It then develops independent cost  estimates for major milestone reviews and in the event that an acquisition  program experiences a Nunn-McCurdy breach.  According to PARCA, it  assesses all major defense acquisition programs at least once per quarter  or when requested by the Under Secretary of Defense for Acquisition,  Technology and Logistics, and disseminates this information to senior  leaders. The office also interacts with specific programs if they experience  a Nunn-McCurdy breach. In these cases, the office assesses program  performance not less than semi-annually until 1 year after it receives a  new milestone approval.
In addition to the new organizational requirements, the Reform Act  requires DOD to ensure that the acquisition strategy for major defense  acquisition programs includes measures to ensure competition or the  option of competition throughout the program life cycle. This could include  strategies such as maintaining two sources for a system (dual-sourcing)  and breaking requirements for supplies or services previously provided or  performed under a single contract into separate smaller contracts  (unbundling of contracts). Major defense acquisition programs are also  required to provide for competitive prototyping—where two or more  competing teams produce prototypes before a design is selected for  further development—prior to Milestone B unless a waiver is properly  granted by the milestone decision authority, and to meet the following  Milestone B certification requirements, including:
Appropriate trade-offs among cost, schedule, and performance  objectives have been made to ensure the program is affordable;
A preliminary design review and formal post-preliminary design review  assessment have been conducted and on the basis of such  assessment the program demonstrates a high likelihood of  accomplishing its intended mission;
Technology has been demonstrated in a relevant environment on the  basis of an independent review and assessment by the Assistant  Secretary of Defense for Research and Engineering;
Reasonable cost and schedule estimates have been developed to  execute, with concurrence of the Director of CAPE, the program’s  product development and production plan;
Funding is available to execute the program’s product development
DOD has completed an analysis of alternatives with respect to the
The Joint Requirements Oversight Council has approved program  requirements, including an analysis of the operational requirements.
The Reform Act also requires the Joint Requirements Oversight Council  to ensure trade-offs among cost, schedule, and performance objectives  are considered for joint military requirements. GAO previously reported  that the Council considered trade-offs made by the military services  before validating requirements, but the military services did not  consistently provide high-quality resource estimates to the Council for  proposed programs in fiscal year 2010. We also found that the Council  did not prioritize requirements, consider redundancies across proposed  programs, or prioritize and analyze capability gaps in a consistent  manner.
section_title: DOD Has Implemented Most of the Fundamental Reform Act Provisions and Is Strengthening Acquisition Activities
paragraphs: DOD has implemented most of the fundamental Reform Act provisions as  required and is taking additional steps to strengthen acquisition reviews,  policies, and capabilities. Offices established as a result of the Reform  Act are continuing to issue policies, review and approve relevant  acquisition documents, monitor weapon acquisition program activities,  and develop performance measures. In addition, all four of the major  defense acquisition programs we reviewed that had not started  development when we selected our case studies plan to implement  Reform Act provisions regarding preliminary design reviews, competitive  prototyping, and competition. Also, some provisions, such as issuance of  guidance on estimating operating and support costs, by the CAPE, are  still in the process of being completed. Finally, we found that the Under  Secretary of Defense for Acquisition, Technology and Logistics has  revised the defense acquisition review process to consider additional  knowledge collected on programs earlier and efforts are being made to  strengthen acquisition policies and capabilities.
subsections: section_title: DOD Is Continuing to Implement Reform Act Provisions
paragraphs: The offices established as a result of the Reform Act—SE, DT&E, CAPE,  and PARCA—are continuing to make progress in implementing four  fundamental Reform Act provisions aimed at strengthening acquisition  outcomes and oversight of weapon acquisition programs. Specifically, the  offices are (1) developing policy and guidance to the military services for  conducting work in their respective areas, (2) approving acquisition  documents prior to milestone reviews, (3) monitoring and assessing  weapon acquisition program activities on a consistent basis, and   (4) developing performance measures to assess acquisition program  activities. Figure 1 provides the status of DOD efforts to implement the  four fundamental provisions. Some offices are still in the process of  completing a few of these provisions. For example, CAPE and PARCA  are in the process of developing policies and guidance for their respective  areas and DT&E is in the process of establishing performance measures  that can be used to assess weapon acquisition program activities. The  office piloted the performance measures on two major defense acquisition  programs and reported that they are currently applying them to over 40  programs. Note that some activities related to approving documents and  monitoring or assessing programs require on-going efforts on the part of  some of the offices.
We also found evidence that major defense acquisition programs are  integrating Reform Act provisions in their acquisition strategies. The four  weapon acquisition programs we reviewed that had not started  development activities when we began our review plan to implement  Reform Act provisions related to preliminary design reviews, competitive  prototyping, and competition. For example, the Ground Combat Vehicle  has two contractors developing competitive prototypes of two key  subsystems to support technology development. The program intends to  conduct preliminary design reviews on both contractors’ designs prior to  Milestone B and to conduct full and open competition through Milestone  C. Similarly, according to program officials, the Joint Light Tactical  Vehicle program had three contractors develop full-system prototypes  during the technology development phase and held preliminary design  reviews on each contractor’s design prior to Milestone B. The program  plans to continue competition throughout engineering and manufacturing  development. None of the four programs in our review received a waiver  from Reform Act provisions.
section_title: Additional Emphasis Is Being Placed on Improving Milestone Review Process, Policies, and Service Capabilities
paragraphs: OSD is taking additional steps to strengthen the department’s oversight of  weapon acquisition programs and guidance for developing the programs.  In June 2011, for example, the Under Secretary for Acquisition,  Technology and Logistics revised the weapons acquisition review process  to consider acquired knowledge on weapon acquisition programs earlier  than before. The revised review process includes two new review points.  The first new review—the pre-engineering and manufacturing  development review—occurs before the release of a final request for  proposal for the engineering and manufacturing development phase. The  purpose of this new review is to assess each program’s acquisition  strategy, request for proposal, and key related planning documents earlier  in the process, and to determine whether program plans are affordable,  executable, and reflect sound business arrangements. The second new  review—the acquisition strategy and request for proposals review and  approval—occurs prior to Milestone C, the production decision. The  review provides the milestone decision authority an opportunity to review  the acquisition strategy and request for proposals for the production and  deployment phase prior to Milestone C. Figure 2 illustrates the revised  review process.
According to the Under Secretary of Defense for Acquisition, Technology  and Logistics, who is the authority for making milestone decisions for  most major weapon acquisitions, the prior review process did not provide  an adequate opportunity for review of program plans prior to release of  the final request for proposals—the point at which DOD’s requirements,  schedule, planned program content, and available funding should be firm  and available for review. Further, the Under Secretary stated that making  changes to acquisition strategies and program plans after all bidding  activities, proposal evaluation, and source selection are complete is  difficult and highly disruptive.
DOD is also rewriting the DOD Instruction 5000.02 to include an  extensive restructure of acquisition policies according to the Under  Secretary of Defense for Acquisition, Technology and Logistics. This  update will implement Section 832 of the National Defense Authorization  Act for Fiscal Year 2012, which requires DOD to issue guidance on  actions to be taken to improve its processes for estimating, managing,  and reducing operation and support cost, as well as ensure competition in  maintenance and sustainment of subsystems of major weapon systems,  among other things. In addition to current policies implementing the  Reform Act, officials stated that key provisions from the Reform Act will  also be included in the updated instruction.
In addition to implementing the provisions of the Reform Act, DOD offices  have taken other steps to strengthen acquisition capabilities throughout  the department. For example:
The SE office, according to DOD officials, led efforts to establish  working groups to help the services address systemic reliability issues  across the unmanned aircraft and rotary wing portfolios earlier in the  process. The office also led several workforce development initiatives  to attract and retain a qualified engineering workforce and supported  the implementation of legislation requiring each acquisition program  office to name a key technical advisor who is responsible for all  engineering activities.
The DT&E office, according to DOD officials, championed updates to  DOD Instructions that will require weapon acquisition programs to  consider using DOD test capabilities before paying contractors to  develop similar capabilities. In addition, the office supported  legislation requiring major defense acquisition program offices to have  a government test agency serving as the lead developmental test and  evaluation organization for the program and a chief developmental  tester. The chief developmental tester position, as included in the  National Defense Authorization Act for Fiscal Year 2012, is tasked  with coordinating the planning, management, and oversight of all  developmental testing activities, among other things.
The CAPE office, according to DOD officials, established an operating  and support cost directorate to build its expertise and place more  emphasis on developing better operating and support cost estimates  throughout the acquisition life cycle. This directorate will coordinate  the development of an operating and support cost estimating  guidebook.
The PARCA office, according to DOD officials, is providing additional  insights to the Under Secretary of Defense for Acquisition,  Technology and Logistics on systemic acquisition problems.  Specifically, the office is examining a wide range of acquisition-related  information from the past 40 years, such as contract type, stability of  key performance parameters, and program manager tenure to  determine if there is any statistical correlation between these factors  and good or poor acquisition outcomes.
section_title: The Reform Act Is Helping Programs Identify and Mitigate Risks Earlier in the Acquisition Process
paragraphs: We identified four key areas where the Reform Act had a significant  influence on programs in the 3 years since it was enacted:   (1) requirements, (2) cost and schedule, (3) testing, and (4) reliability.  These four areas have been common sources of problems in the past.  For example, the services typically started new weapon acquisition  programs with requirements that were both demanding and inflexible and  planned to use relatively unproven technologies to meet the  requirements—all of which increased program risks. In addition, cost and  schedule estimates were frequently too optimistic based on the proposed  requirements and technologies. Design problems stemming from rigid  requirements and the use of immature technologies to meet them were  often discovered during testing and fixed late in the development cycle  and resulted in cost increases, performance shortfalls, and schedule  delays. Finally, DOD’s inattention to reliability has resulted in a dramatic  increase in the number of systems that have not met suitability  requirements during operational testing. Deficiencies—such as high  failure rates and disappointing improvements in the reliability, availability,  and maintainability of weapon systems—have limited program  performance and increased operation and support costs. We examined  11 programs at various stages of the acquisition process to determine  how the offices and policies established as a result of the Reform Act  impacted their acquisition strategies and decision-making process. Four  programs had not yet passed Milestone B, development start, at the time  we began our review. Of the remaining seven programs, three had  breached Nunn-McCurdy cost thresholds since the act was passed and  have had to satisfy the Reform Act’s new requirements with regards to  certification. The other programs had significant interaction with one or  more of the OSD offices established by the Reform Act. Table 3 indicates  in which of the four areas each program has been affected by the Reform  Act.
In some cases, programs have made changes based on input from OSD  offices like systems engineering or developmental test and evaluation; in  other cases, programs have integrated Reform Act policies, such as  preliminary design reviews and competitive prototyping, into their  acquisition planning. Programs that were already in development or  production when the Reform Act was passed were less likely to have  interactions with the OSD offices on requirements trades because these  discussions typically occur prior to Milestone B. A discussion of how  individual programs have been affected in the areas of requirements, cost  and schedule realism, testing, and reliability follows.
subsections: section_title: Greater Emphasis Is Being Placed on Requirements, but Challenges Remain
paragraphs: The Reform Act places significant emphasis on early problem solving and  requires programs to put much more effort toward considering trade-offs  among cost, schedule, and performance requirements prior to Milestone  B. As part of this effort, it requires the Secretary of Defense to ensure that  acquisition, budget, and cost estimating officials have the opportunity to  raise cost and schedule matters before performance objectives are  established. The Reform Act also charges the Joint Requirements  Oversight Council with the responsibility to ensure that cost, schedule,  and performance trade-offs for joint military requirements are considered,  and to include combatant commanders in the process to ensure the  user’s needs are adequately satisfied. The offices established as a result  of the Reform Act have helped programs, such as the Joint Light Tactical  Vehicle and Ground Combat Vehicle, make trade-offs among cost,  schedule, and technical performance requirements. As a result, these  programs have developed a more realistic acquisition strategy from a  cost, schedule, and technical standpoint.
Joint Light Tactical Vehicle: The program held several reviews prior  to Milestone B to identify, modify, or eliminate requirements that were  unachievable or unaffordable, thus establishing a more technically  realistic program. Officials from DT&E and SE participated in these  reviews. By involving both the requirements and acquisition  communities in the reviews, the Army was able to reduce the required  capability to cut costs while ensuring that trade-off decisions would  not impair the system’s ability to meet the warfighter’s operational  needs. Examples of requirements changes that helped to cut costs as  well as reduce risk include: allowing the active suspension system, crew displays, and  integrated starter-generator to be tradable design features. These  changes resulted in a 30 percent reduction in the average unit  manufacturing cost from the initial target of $475,000 to $331,000,  while at the same time reducing technical and weight risk.  According to program officials, this makes the $250,000 unit  manufacturing goal more achievable.   reducing the reliability requirement and changing the Army  helicopter lift requirement based on the results of technology  development prototype testing. This mitigated technical risks  going into development.
The program recently moved into engineering and manufacturing  development, but not all requirements issues have been resolved and  future trade-offs may be necessary. For example, early testing  showed that none of the three prototype variants met the program’s  soft soil or sand slope requirement. This requirement has not been  changed. Program and OSD officials are monitoring this issue closely  and plan to actively manage it during engineering and manufacturing  development.
Ground Combat Vehicle: The Ground Combat Vehicle program  exhibited some of the same problems experienced by previous DOD  programs prior to Milestone B—demanding and inflexible  requirements. The SE office and the Under Secretary of Defense for  Acquisition, Technology and Logistics are helping the program set  achievable requirements. Following its materiel development decision  in February 2010, the program issued a request for proposals that  contained nearly 1,000 requirements and a challenging 7-year  schedule for the delivery of the first production vehicle. At the request  of the Under Secretary for Acquisition, Technology and Logistics, the  Army established an independent review team to assess the risks  associated with the program’s schedule. The team, which included an  SE official, raised concerns about the program’s high number of  mandatory requirements and the risks associated with the 7-year  schedule. To mitigate program risks, the Army reduced the number of  performance requirements by about 25 percent and prioritized the  others, giving competing contractors flexibility in addressing some  performance requirements. The Army issued a revised request for  proposal in November 2010. In August 2011, the Under Secretary of  Acquisition, Technology and Logistics approved the program’s entry  into technology development, but expressed concern about the cost  and schedule risks associated with delivering a production vehicle in 7  years. Because of these concerns, the Under Secretary directed the  Army to consider other alternatives, such as existing vehicles, that  could meet warfighter needs. The analysis is currently planned to be  completed in March 2013 to inform the Milestone B decision.
section_title: Program Cost and Schedule Estimates Are Becoming More Realistic
paragraphs: By establishing a new cost assessment and program evaluation office  and requiring this office to scrutinize program cost and schedule  estimates beginning at Milestone A, CAPE officials believe that the  Reform Act has helped infuse more realism in cost estimates and  promote earlier discussions about affordability. CAPE officials also  believe that because their independent cost and schedule estimates have  become more visible within DOD and Congress, the military services are  developing more realistic estimates. We saw evidence of these benefits in  the programs we reviewed, including the Ohio Class Replacement,  Littoral Combat Ship Seaframe, and F-35 Joint Strike Fighter programs.
Ohio Class Replacement: The CAPE office was involved in the  decision-making process to ensure program affordability. The office  prepared an independent cost estimate and reviewed the program’s  affordability goals prior to Milestone A. The service and independent  estimates were within 2 percent of each other. However, the Under  Secretary of Defense for Acquisition, Technology and Logistics  directed the Navy to do a rigorous cost comparison of a 16 missile  tube design versus a 20 missile tube design. The Navy determined  that a 16 missile tube configuration would meet warfighter  requirements and users’ needs while reducing program costs by  about $200 million per submarine, or approximately $3 billion for the  total program. It would also simplify the ships’ design and integration  effort. The CAPE office validated the savings associated with the 16  missile tube design. As a result, the Navy incorporated the 16 missile  configuration as the program baseline.
Littoral Combat Ship Seaframe: The CAPE office helped make  program costs more visible. Prior to the program’s Milestone B  decision, CAPE completed an independent cost estimate of the  seaframe program and found that the resources in the future years’  defense plan budget were lower than the projected program costs for  the same time period. Navy officials attributed this problem to the  overlap between the timing of the milestone decision and the  president’s budget submission. The office further noted that the  resources in the Navy’s budget did not include the additional  development activities required to support two full ships. Without this  information, decision makers would not have had visibility into the  expected costs of the seaframe program or be able to make more fully  informed decisions. As a result, the Navy re-phased its funding in the  budget, adding approximately $397 million to fully fund the  development program.
Joint Strike Fighter: The SE and CAPE offices helped the program  develop more realistic cost and schedule estimates. CAPE officials  have been involved in reviews of the F-35 Joint Strike Fighter program  even prior to the passage of the Reform Act and have continued to be  heavily involved in subsequent programs reviews. For example, the  cost analysis improvement group, which was the predecessor to  CAPE, led a multi-functional joint estimating team review of the  program in 2008. This review found problems with the program’s  funding and schedule. In 2010, the Joint Strike Fighter program  notified Congress that its estimated unit costs had increased by more  than 80 percent since the original Milestone B baseline in 2001. This  increase triggered a Nunn-McCurdy unit cost breach and later  prompted the program executive officer to commission a technical  baseline review of the program to help determine the resources  needed to complete development. Officials from the SE office  participated in this technical review. The CAPE office also did an  independent cost estimate of the program as part of the Nunn- McCurdy certification process. Based on information from these  efforts, DOD developed a more realistic program plan by adding $4.6  billion to the development program, reducing near-term procurement  quantities by 125 aircraft, and extending the development test period  by 4 years to accommodate developmental testing, address the  increased program scope, and fix software issues.
section_title: Programs Are Adding More Tests to Reduce Development Risks
paragraphs: The Reform Act significantly strengthened the role of developmental  testing in the department. In the 2 decades prior to the Reform Act, the  prominence of developmental testing had declined within OSD. In the  early 1990’s, developmental testing was part of an all-encompassing test  organization that reported directly to the Under Secretary of Defense for  Acquisition. According to a former senior developmental testing official, by  2004, two people worked on developmental testing activities within the  systems engineering organization. In establishing a separate office for  DT&E, the Reform Act reinforced the need for robust developmental  testing early in the acquisition process. The Reform Act gave the Deputy  Assistant Secretary for Developmental Test and Evaluation formal  approval authority for the test and evaluation master plans of major  defense acquisition programs. This authority enables the Deputy  Assistant Secretary to help ensure that programs have robust test and  evaluation plans. Our case study reviews illustrate the efforts that the  DT&E office has made to help programs such as the Small Diameter  Bomb II and KC-46 Tanker obtain more design and performance  knowledge early in a program’s acquisition life cycle.
Small Diameter Bomb II: Air Force program officials acknowledged  that developmental and operational testing officials worked closely  with them as they prepared the test and evaluation master plan for the  Milestone B decision. After reviewing the plan, developmental and  operational test officials concluded that the program would benefit  from adding a 28-shot test program prior to entering operational  testing. According to the program office, the purpose of the additional  testing is to further establish the performance of the weapon in  realistic scenarios and to increase the likelihood of completing  operational testing without a failure. DT&E officials stated this testing  would provide more complete knowledge about the bomb’s  functionality and help reduce risk of a major redesign moving forward.  Program officials stated that they allocated an additional $41 million to  its developmental test program to conduct the 28 additional flight tests  prior to operational testing.
KC-46 Tanker: The program office acknowledged that the DT&E  office, as part of an integrated test team comprised of government  and industry officials, helped identify options that could add time to the  test plan for important testing if unexpected delays are encountered.  In its fiscal year 2011 annual report, the operational test and  evaluation office reported that the KC-46’s planned flight test program  was not executable, determining that more time would be needed for  military flight-testing. It based this conclusion on the historical flight  test experience of similar programs. Program officials stated that they  were initially reluctant to change the test plan because they had  awarded a fixed price contract and any changes could result in  reopening the contract, leading to potential cost increases. However,  the integrated test team identified a recovery period that may be  applied to the KC-46 aerial refueling certification if delays are  encountered. The contractor now has a plan that could allocate an  additional 1.5 months for two test aircraft to complete this testing, if  deemed necessary. This testing would provide more knowledge about  the program’s aerial refueling performance prior to operational testing.  DT&E officials stated that they plan to continue working with the  program to address overall flight test challenges. While testing  remains one of the program’s risk areas, this change may lessen that  risk.
section_title: Greater Emphasis Is Being Placed on Weapon System Reliability
paragraphs: The Reform Act emphasizes the need for designing more reliable weapon  systems. It charges the Deputy Assistant Secretary for Systems  Engineering with the responsibility to ensure the systems engineering  approach used by major acquisition programs includes a robust plan for  improving reliability. The DT&E office reviews programs’ reliability growth  test plans. This testing provides visibility over how reliability is improving  and uncovers design problems so fixes can be incorporated before  production begins. A reliability growth curve is used to track projected and  actual improvements in reliability over time. The Reform Act further  requires that the Deputy Assistant Secretary for Systems Engineering  develop policies and guidance for the inclusion of provisions relating to  systems engineering and reliability growth in requests for proposals. We  observed evidence of this increased emphasis in the Joint Light Tactical  Vehicle, Remote Minehunting System, Gray Eagle, and Global Hawk  programs.
Joint Light Tactical Vehicle: The DT&E office helped this program  develop a more realistic reliability growth plan prior to Milestone B.  Based on the performance of prototype vehicles, developmental test  officials determined that the program’s reliability growth curve was  unrealistic. For example, officials reported that the program’s initial  reliability growth plan assumed a starting reliability that was almost 60  percent higher than what had actually been demonstrated during  technology development. It also assumed commonality between the  two vehicle variants, a large reliability increase in a short test time,  and two corrective action periods. The DT&E office recommended  that the program eliminate the vehicle commonality assumption, add  more test miles, and add another corrective action period to its test  plan. It also recommended that the program consider lowering the  vehicle’s reliability requirement. Based on this input, the Army revised  its plan by adding two vehicles and 40,000 more test miles to ensure  reliability is adequately addressed for both variants. With approval of  the user, it also reduced the reliability requirement from 3,600 to 2,400  miles mean time between operational mission failures.
Remote Minehunting System: The SE office worked with program  officials to improve reliability growth planning, which was found to be  one of the key factors leading to the program’s Nunn McCurdy unit  cost breach in 2009. Before the breach, program officials had not  funded a reliability growth program or established a design for  reliability process. The program had a reliability goal of 150 hours  mean time between failures, but program officials stated that testing  demonstrated a reliability of only about 45 hours. Since the breach,  the program has worked closely with the SE office to establish a  reliability program plan and a growth curve to track reliability  improvements. During the Nunn-McCurdy review, the program  developed a three phase reliability growth program to improve the  program’s subsystems, components, and manufacturing processes  that contributed to poor reliability. According to program officials,  phase one of the reliability growth program was completed in 2011,  and reliability has improved by 40 percent, going from 45 hours mean  time between operational mission failures to 63 hours. Although this  improvement is still below the minimum requirement of 75 hours,  program officials stated that phase two of the reliability growth  program is scheduled to be completed in April 2013 and is projected  to achieve the program’s 75 hour minimum requirement.
Gray Eagle and Global Hawk: The SE office has worked to improve  reliability across the unmanned aircraft portfolio, including the Gray  Eagle and Global Hawk. Prior to the Gray Eagle’s second low rate  initial production decision in 2011, SE officials raised concerns about  the system’s poor reliability. As a result, the Army was directed to  undertake a reliability improvement program. The Under Secretary of  Defense for Acquisition, Technology and Logistics approved the  program for low rate initial production, but stressed the need to  improve the operational reliability as quickly as possible. SE officials  worked with program officials to establish a reliability working group,  develop reliability growth curves, and develop a reliability  enhancement management plan. According to SE officials, the Gray  Eagle initially improved the reliability of the aircraft by 15 percent and  the ground control station by 30 percent. According to program  officials, the initial reliability goals were overstated and not needed to  meet the program’s overall operational availability requirement. Based  on initial operational test results in August 2012, the program office is  working with the user to redefine the reliability goals without impacting  the system’s ability to meet its overall operational availability  requirement. According to the PARCA office, these efforts have been  informed by a detailed reliability model that they built in consultation  with the Army. This model showed the relationship between the  aircraft’s reliability and its availability to perform operational missions.  SE officials also found similar reliability problems on the Global Hawk  program and worked with program officials to establish a reliability  growth and improvement plan and reliability growth curves. According  to SE officials, the time between unscheduled maintenance on the  Global Hawk has improved on the order of 50 to 80 percent.
section_title: Challenges Exist That Could Limit the Reform Act’s Ability to Influence Systemic Change
paragraphs: While DOD has taken steps to implement most of the fundamental  Reform Act provisions, some key efforts to date have been primarily  focused on DOD’s largest major defense acquisition programs.  Expanding the reach of the Reform Act to bring about systemic change to  DOD’s weapons acquisition process so that it influences all programs,  however, still has challenges. Although senior leaders were receptive to  the Reform Act principles, they identified several challenges that currently  limit DOD’s ability to broaden the Reform Act’s influence. We grouped  these challenges into five general categories: (1) organizational capability  constraints; (2) need for additional guidance on cost estimating and  Reform Act implementation; (3) uncertainty about the sufficiency of  service level systems engineering and developmental testing resources;  (4) limited dissemination of lessons learned; and (5) cultural barriers.
subsections: section_title: Organizational Capability Constraints Make It Difficult to Expand Reform Act’s Impact
paragraphs: Leaders of two of the offices established as a result of the Reform Act told  us that even though they have implemented most of the fundamental  Reform Act provisions, they have had to limit their activities to a portion of  acquisition programs in their portfolios due to resource constraints. Thus,  it is doubtful that they could expand the scope of their activities to include  more weapon acquisition programs at current staffing levels. For  example, the DT&E office has had to be selective in its level of oversight  of acquisition programs because the current staff of around 70  government and contractor personnel cannot adequately cover a portfolio  of over 200 acquisition programs, according to its Deputy Assistant  Secretary. The office has dropped virtually all but the largest programs  from its oversight list and eliminated oversight of some major automated  information systems. CAPE officials estimated that its cost assessment  division would need to double in size in order to meet the Reform Act’s  requirements. However, soon after the Reform Act was enacted,  budgetary constraints limited the expansion of the cost estimating  workforce to about 25 percent of the necessary growth. According to  CAPE officials, its current cost analysis staffing is not adequate to meet  its mission of improving the analytical skills of the defense cost estimating  workforce, issuing policy, and providing sound and unbiased cost and  schedule estimates. The office has delegated its independent cost  estimating responsibility for most major automated information systems to  the military services and some guidance has yet to be issued.
The SE and PARCA offices are also struggling in some regards. For  example, according to its Deputy Assistant Secretary, the SE office is  continuously challenged to maintain the high caliber, qualified personnel  required to provide assistance to and oversight of its portfolio of over 200  acquisition programs. Further, PARCA officials stated that the availability  of government positions, particularly at the senior executive service level,  continues to be a critical issue for the office. The two divisions within the  PARCA office, the performance assessments division and the root cause  division, do not currently have permanent government personnel at the  senior executive level. Officials also stated current proprietary information  rules limit the ability of PARCA contractor personnel to handle and  maintain some weapon system information, severely impeding  operations.
section_title: Services Would Like More Cost Estimating and Reform Act Implementation Guidance
paragraphs: Offices within OSD have not yet issued more detailed guidance that could  help institutionalize better cost estimating practices and steer program  decisions related to competitive prototyping and preliminary design  reviews. The CAPE office has not issued guidance for operating and  support costs estimates, such as fuel and maintenance costs, that have  been estimated to account for two-thirds or more of a system’s total life  cycle cost.  In addition, although not specifically required by the Reform  Act, the CAPE office has not issued guidance for the services to use  when developing Milestone A program cost estimates.  As a result, senior  leaders may not have access to realistic cost estimates prior to Milestone  B for decision making purposes. Military service officials told us they are  particularly interested in getting guidance on data that should be included  in the cost analysis requirements description, which forms the basis for  their cost estimates at Milestones B and C. CAPE officials recognize that,  while some progress has been made, they need to complete the  guidance, but have not been able to dedicate resources to do so.
Some officials also told us that they find the competitive prototyping and  preliminary design review requirements confusing and would like  guidance on how to implement these requirements. DOD policy requires  the technology development strategy for major defense acquisition  programs to provide for prototypes of the system or, if a system prototype  is not feasible, for prototypes of critical subsystems before Milestone B  approval. However, officials from the Ground Combat Vehicle program  were unclear as to when and what type of prototype to use. From a  broader perspective, other military officials questioned the value of  competitive prototyping as a blanket requirement for all programs,  especially for programs that are using mature technologies, given the  cost. For example, senior acquisition officials questioned the necessity of  spending $400 million on competitive prototyping for the Small Diameter  Bomb II program since the program was aware of problems with one  contractor’s design. However, program officials indicated that competitive  prototyping enabled them to identify design issues early in development  and realize a savings of $1 billion. Officials from the Ground Combat  Vehicle program we spoke with also indicated that they struggled with the  timing of when to hold the program’s preliminary design review and what  type of knowledge was required, since better guidance is needed. The  program plans to hold multiple design reviews prior to Milestone B to  consider contractor and government designs of the weapon system and  then hold another review after Milestone B in order to resolve differences  between the government’s and selected contractor’s preliminary designs.
We spoke with OSD officials to determine which office should be  providing guidance or assistance to program managers on competitive  prototyping and preliminary design review issues. None of the offices  have official responsibility for these efforts. OSD officials stated that these  are program decisions and should be discussed with their respective  military service level acquisition officials.
section_title: Services May Not Have Sufficient Resources to Oversee and Conduct Systems Engineering and Developmental Testing Activities
paragraphs: OSD officials believe that the services may lack resources in key  positions that could help strengthen systems engineering and  developmental testing activities on weapon acquisition programs. For  example, according to the Deputy Assistant Secretary of Defense for  Systems Engineering, the Navy and Air Force have reassigned the duties  and responsibilities of their service-level chief engineers, thereby de- emphasizing the importance of systems engineering. The Deputy  Assistant Secretary believes maintaining strong systems engineering  leadership at the service level is essential for tying the systems  engineering community together and promoting good systems  engineering practices throughout each respective service. According to  the DT&E and SE office’s March 2012 joint annual report to the  Congress, the Navy abolished its chief engineer position and while the Air  Force recently began to take steps to relocate the systems engineering  function to the headquarters level, the impact of a recent reorganization  on systems engineering activities is not yet known. In addition, the Deputy  Assistant Secretary for Developmental Test and Evaluation expressed  concern that the military services may not be implementing new  legislation that requires each major defense acquisition program be  supported by a chief developmental tester that oversees developmental  test and evaluation activities. He stated that in some cases one person is  serving as the chief developmental tester across multiple programs  instead of having one person dedicated specifically to each program.  The Deputy Assistant Secretary is trying to determine the extent to which  this practice is occurring and then plans to work with the services to get  more focused leadership for each program.
It is also unclear whether the services have a sufficient number of  qualified personnel to conduct systems engineering and test and  evaluation activities. The services planned to grow these workforces by  over a combined 5,000 people between fiscal years 2009 and 2015 and  had made progress in growing each of these workforces through fiscal  year 2010. However, budget cuts have resulted in DOD canceling some  of its weapon acquisition programs and reassessing its decision to  increase the acquisition workforce. Last year, we recommended that the  Secretary of Defense report the impact budget cuts were having on the  military service workforce and their ability to meet weapon acquisition  program needs in the areas of developmental testing and systems  engineering. In the DT&E and SE offices’ March 2012 joint annual report  to the Congress, the Deputy Assistant Secretary for Systems Engineering  reported that the Army has reduced its systems engineering workforce  growth plan as compared to the plan reported in March 2011 joint annual  report, and that contractor-to-civilian conversions have been suspended.  In addition, the Deputy Assistant Secretary believes a prolonged hiring  freeze in the Air Force could potentially create new experience gaps in  the workforce. The Deputy Assistant Secretary for Developmental Test  and Evaluation did not discuss the impact of budget cuts on the services’  test and evaluation workforce growth plan in the March 2012 joint annual  report to the Congress and neither office reported on whether the  services had an adequate workforce to meet the needs of the current  portfolio of weapon acquisition programs.
section_title: Expanded Use of Lessons Learned Would Help Expand Impact across the Acquisition Portfolio
paragraphs: DOD has not taken full advantage of sharing lessons learned obtained  through root cause analyses of programs that experience Nunn-McCurdy  cost and schedule breaches with the acquisition workforce, particularly  program managers. According to the Defense Acquisition Guidebook,  which provides best practices the acquisition workforce can use on  programs, lessons learned are a tool that the program manager may use  to help identify potential areas of risk associated with a weapon  acquisition system by reviewing the experiences encountered in past  programs. Lessons learned databases document what worked and what  did not work in past programs, in the hopes that future programs can  avoid the same pitfalls. Further, if the right best practices are applied,  they help to avoid common problems and improve quality, cost, or both.
The PARCA office has made some effort to educate program managers  on how to avoid acquisition problems through classes taught at the  Defense Acquisition University. However, these courses are geared  towards educating new program managers and may not be reaching a  wide range of program officials. Nevertheless, some officials indicated  that this information would be helpful for program officials to understand  and avoid problems that have affected weapon acquisition programs in  the past. Other officials also stated that it would be helpful if root cause  analysis assessments contained more detailed information so acquisition  officials could better understand problems and apply lessons learned. For  example, when cost estimating was determined to be a root cause of a  problem, officials stated they would have found it more beneficial to know  if immature technologies or unrealistic requirements were the basis for  the poor cost estimate.
section_title: Systemic Changes Will Be Difficult Until Cultural Barriers Are Addressed
paragraphs: Perhaps the most difficult challenge the department faces in making  systemic changes to the acquisition process is changing the cultural  relationship between the military services, which fund and develop new  weapon acquisition programs, and OSD offices, which provide advice to  and oversee the programs. Senior military service officials have told us  they believe they understand and can manage the risks of specific  weapon acquisition programs without much assistance from OSD. On the  other hand, OSD officials believe more assistance is needed, as  evidenced by the high number of programs that have experienced Nunn- McCurdy breaches and poor operational testing results. For example,  since it was established in 2009, the DT&E office has assessed whether  15 programs were ready to begin operational testing. The office  recommended that 5 of the programs—Global Hawk Block 20/30,  Standard Missile 6, Joint Tactical Radio System Handheld Manpack  Small Form (HMS) Rifleman Radio, Joint Tactical Radio System HMS  Manpack, and MQ-1C Gray Eagle—not proceed into operational testing.  However, military service acquisition chiefs decided to allow all 5 of these  programs to proceed anyway. Four of the programs—Global Hawk Block  20/30, Standard Missile 6, Joint Tactical Radio System HMS Rifleman  Radio and Joint Tactical Radio System HMS Manpack—demonstrated  poor performance in operational testing, in areas such as reliability,  effectiveness, or suitability. Operational testing results for the MQ-1C  Gray Eagle have not yet been reported.
On the other hand, a few service officials we met with were reluctant to  accept some recommendations made by OSD offices because they  believed the recommendations were overly burdensome and could  significantly impact weapon acquisition programs’ cost and schedule  outcomes without a lot of benefit. This was the case for the KC-46 Tanker  program, where program officials were concerned that additional testing  recommended by the developmental test and evaluation, and operational  testing offices, as part of the integrated test team, could have significant  contractual implications during development. In this case, officials  identified additional flight test opportunities without having to renegotiate  the fixed price contract. However, the additional allotted test time is not  equivalent to the 6 to 8 months the developmental testing office felt  should be added. A similar situation occurred on the Ship to Shore  Connector program. The Navy disagreed with a DT&E office  recommendation to conduct full system testing prior to procuring  additional craft during initial production. The DT&E office believed the  program was high risk because the Ship to Shore Connector was a  complete redesign of a previous system with no reuse of any major  component (engines, gearboxes, hydraulics, command and control  software). Navy officials, however, believe the program is low risk since it  is an evolutionary program and has one critical technology, a fire  suppression system, which has already been sufficiently demonstrated  and qualified through test and evaluation. In addition, the Navy estimated  that it would cost $15 million to revise the existing production schedule to  accommodate the full system testing as recommended by the DT&E  office. The DT&E office and the Navy reached a compromise whereby  OSD would review available system test results before more craft are  authorized.
section_title: Conclusions
paragraphs: Current fiscal pressures, along with the threat of more to come, have  DOD officials looking for ways to increase buying power by controlling  cost and schedule overruns on weapon acquisition programs. The offices  established as a result of the Reform Act, as well as policy provisions  have helped DOD make inroads towards putting weapon acquisition  programs on more solid footing. Together, the offices and policy  provisions place more attention on requirements, costs, testing, and  reliability as early as Milestone A. The provisions of the act, when  specifically focused on newer programs, are having a positive impact on  the programs and the acquisition process. They show that expert  attention to the cost and achievability of capability requirements, the  assumptions made for cost and funding of programs, and the amount of  systems engineering knowledge that is brought to bear early make  programs more executable.
Although senior officials we spoke with throughout the department are  receptive to the broad principles of the Reform Act, it is too early to tell if  the Reform Act is going to result in systemic change to DOD’s weapon  acquisition process. DOD faces several challenges that must be  addressed to get lasting change—organizational capability constraints,  the need for additional cost estimating and implementation guidance, the  possibility of insufficient systems engineering and developmental testing  resources, limited dissemination of lessons learned, and cultural barriers  between OSD and the services. Some challenges appear to be straight  forward to address, such as providing guidance for estimating operating  and support costs, providing additional guidance for conducting  preliminary design reviews and competitive prototyping activities, and  disseminating lessons learned to the broader acquisition community.  However, they may require more resources, which have been difficult to  obtain.
For Reform Act policies and practices to have a systemic effect across  the entire portfolio of weapon system acquisition programs, the  department must also address challenges related to systems engineering  and developmental testing resources and cultural barriers between OSD  and the services. This begins with the services identifying key leaders at  the headquarters level and within program offices to guide systems  engineering and developmental testing efforts and then ensuring that  there are enough trained staff to carry out these activities. OSD will need  to continue monitoring the services’ efforts. It will also require an  environment where the services stop proposing new weapon systems  with inflexible requirements, immature technologies, and cost, schedule,  and funding assumptions that are too optimistic at the start of a program.  Breaking down cultural resistance to change will take more cooperation  between the Under Secretary for Acquisition, Technology and Logistics  and other OSD offices, and service acquisition executives to address, as  well as continuity of leadership. Efforts by the PARCA office to identify  factors that correlate to good or poor acquisition outcomes, particularly as  it relates to program manager tenure, will be beneficial. The services’  ability to demonstrate that the Reform Act is influencing all weapon  acquisition programs, not just the biggest, will be a key indicator for  determining whether the Reform Act has had a positive effect on DOD’s  culture.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense take the following four  actions to enable systemic change across the entire portfolio of weapon  acquisition programs: direct the Director of Cost Assessment and Program Evaluation to  issue guidance for estimating weapon acquisition program costs at  Milestone A and operating and support costs throughout the  acquisition life cycle by the end of fiscal year 2013 and ensure that the  office prioritizes its resources accordingly to accomplish this task; designate responsibility for providing advice and guidance to program  offices on competitive prototyping and preliminary design reviews to  the appropriate organization within the Office of the Under Secretary  of Defense for Acquisition, Technology and Logistics and ensure that  the guidance is developed. The office(s) designated would be the  focal point for addressing program office issues related to the practical  implementation of these Reform Act provisions, such as the type of  competitive prototyping to use, the timing and benefits of holding  preliminary design reviews prior to milestone B, and if a preliminary  design review should be held after milestone B; direct the Deputy Assistant Secretaries of Defense for Systems  Engineering and Developmental Test and Evaluation to assess and  include in their annual report to the Congress beginning with the  report on fiscal year 2012 activities: the extent to which the office can perform their required activities  with allocated resources;  the impact budget cuts are having on the military services total  workforce (civilians, military, and contractors) and ability to meet  program office needs; and progress the services have made filling leadership positions, such  as chief engineers at the service level and technical leads for  systems engineering and developmental testing at the program  office level; direct the Director of Performance Assessments and Root Cause  Analyses to make lessons learned collected during its root cause  analysis evaluations available to the acquisition workforce and ensure  that the office prioritizes its resources accordingly.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us written comments on a draft of this report. DOD  concurred with two recommendations and partially concurred with two  others. DOD’s comments appear in appendix III. DOD also provided  technical comments, which we incorporated as appropriate in the report.
DOD agreed with the intent of our first recommendation, but noted that  due to resource constraints, the Cost Assessment and Program  Evaluation office could not guarantee that it would be able to issue  guidance for estimating major defense acquisition program costs at  Milestone A and operating and support costs throughout the acquisition  lifecycle by the end of fiscal year 2013. We continue to believe that the  Cost Assessment and Program Evaluation office should issue the  guidance by the end of fiscal year 2013. However, if that is not possible  from a resource standpoint, the office should commit to a date and devote  the resources to meeting that date. We will continue to monitor DOD’s  efforts to develop the guidance.
Although DOD concurred with our second recommendation, we revised  this recommendation based upon discussions with DOD officials during  the agency comment period. Our revision clarified the intent of this  recommendation, which is to have the Secretary of Defense designate a  specific organization within the Office of the Under Secretary of Defense  for Acquisition, Technology and Logistics to provide advice and guidance  on competitive prototyping and preliminary design reviews. We  understand that the department has issued Reform Act implementation  guidance and has incorporated aspects of competitive prototyping and  preliminary design reviews in the Defense Acquisition Guidebook.   Further, we recognize that program offices we visited are taking steps to  implement the guidance that has already been issued. However, based  on our discussions with senior level officials, we believe one or more  offices need to be designated with the responsibility of developing  additional guidance and answering program specific questions related to  the practical implementation of the requirements.  As noted earlier in our  report, some officials questioned when to use prototyping or what type of  prototyping should be used. In addition, there were questions about the  timing of the preliminary design reviews.
DOD partially concurred with our third recommendation. DOD noted that  the type of information we recommended be assessed and reported on   should be included as part of DOD’s human capital strategic planning  process and as such, be reported in DOD’s annual Acquisition Workforce  Strategic Plan. We agree that the impact of budget cuts on the workforce  and the status of leadership positions could be addressed in the annual  strategic plan. However, we continue to believe that the Deputy Assistant  Secretaries for Systems Engineering and Developmental Test and  Evaluation should include an assessment in their joint annual report to the  Congress on the respective offices’ ability to perform activities specified in  the Reform Act with available resources.
DOD concurred with our fourth recommendation, which would make  lessons learned from root cause analyses available to the acquisition  workforce.
We are sending copies of this report to the Secretary of Defense and  appropriate Congressional Committees. In addition, the report is available  at no charge on the GAO website at http://www.gao.gov.
If you have any questions about this report or need additional information,  please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact  points for our Office of Congressional Relations and Public Affairs may be  found on the last page of this report. Key contributors to this report were  Cheryl Andrew, Assistant Director; Laura Greifner, Julie Hadley, Megan  Porter, Rae Ann Sapp, and Marie Ahearn.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report examines DOD’s continued implementation of the Weapon  Systems Acquisition Reform Act of 2009 (Reform Act). Specifically, we  examined (1) DOD’s progress in implementing Reform Act provisions;   (2) the impact the Reform Act has had on specific acquisition programs;  and (3) challenges remaining in improving the weapons acquisition  process.
To assess DOD’s progress in implementing Reform Act provisions, we  interviewed officials and analyzed documents, such as reports to the  Congress and guidance issued from the Office of the Secretary of  Defense (OSD) offices of the (1) Deputy Assistant Secretary of Defense  for Systems Engineering (SE), (2) Deputy Assistant Secretary of Defense  for Developmental Test and Evaluation (DT&E), (3) Cost Assessment and  Program Evaluation (CAPE), and (4) Performance Assessments and  Root Cause Analyses (PARCA) to determine the extent to which  provisions have been implemented. We focused our review on the offices’  implementation of four fundamental Reform Act provisions: developing  policy and guidance; approving acquisition documents; monitoring  programs and conducting program assessments; and developing  performance measures. In cases where provisions had not been  implemented, we asked officials about the reasons for the delay and the  expected time frame for completion. We also interviewed officials and  analyzed documents from the office of the Assistant Secretary of Defense  for Research and Engineering and the Defense Procurement and  Acquisition Policy office, as well as four weapon acquisition programs,  which had not yet started development to determine the progress DOD  has made implementing Reform Act provisions related to preliminary  design reviews, competitive prototyping, and competition. We believe  these programs offer the best glimpse at how the OSD offices and  Reform Act policies are influencing acquisition strategies. The weapon  acquisition programs we chose for this analysis were part of a larger case  study review that is described below.
To determine the impact the Reform Act has had on specific weapon  acquisition programs, we selected 11 weapon system programs to use as  case studies. For each program we reviewed relevant program  documentation such as the test and evaluation master plans,  assessments of operational test readiness, systems engineering plans,  program support reviews, root cause analyses, analysis of alternatives  reports and cost estimates as applicable. We also interviewed appropriate  program officials and officials from the OSD offices for SE, DT&E, and  CAPE to obtain their perspectives about (1) the level of interaction  between the programs and OSD offices; (2) changes made to program  acquisition strategies as a result of interactions with the OSD offices; and  (3) benefits and challenges with implementing Reform Act provisions on  each of the programs. We also reviewed the performance assessments  and root cause analyses office’s root cause analysis documentation for  programs that incurred Nunn-McCurdy cost or schedule breaches.
We selected our case studies based on input from the officials in the OSD  offices for SE, DT&E, CAPE, PARCA, and operational test and  evaluation. We also discussed possible case studies with GAO  employees who monitor and report on weapon acquisition programs on  an annual basis. The programs we selected for review represent a variety  of platforms, including sea vessels, manned and unmanned aircraft, and  land systems. Specifically, we examined 11 programs at various stages of  the acquisition process. Four programs had not yet passed Milestone B,  development start, at the time we began our review. The remaining seven  programs had completed their Milestone B review and were in  development at the time of our case study selection. Of the seven  programs, three have breached Nunn-McCurdy cost thresholds since the  act was passed and have had to satisfy the act’s new requirements with  regards to certification. The other programs had significant interaction  with one or more of the OSD offices established by the Reform Act. A  complete list of programs is provided below.
While our sample of 11 case studies allowed us to learn about the impact  the Reform Act offices have had on DOD acquisitions, it was designed to  provide anecdotal information, not findings that would be representative  of all the department’s weapon acquisition programs.
To determine challenges remaining in improving defense acquisitions we  relied on information we collected and analyzed during our case study  review of 11 weapon acquisition programs. We also solicited the opinions  of the Under Secretary of Defense for Acquisition, Technology and  Logistics; other senior level officials in the Office of the Secretary of  Defense including the leaders of each of the offices created as a result of  the Reform Act; as well as the military services’ Senior Acquisition  Executives.
We conducted this performance audit from January 2012 to December  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Progress of Reform Act Offices in Implementing Weapon System Acquisition Reform Act Provisions
paragraphs: Appendix II: Progress of Reform Act Offices in  Implementing Weapon System Acquisition  Reform Act Provisions Status  Complete: Issued a reliability, availability, and maintainability Directive-Type  Memorandum, a development planning Directive-Type Memorandum, and a DOD  Instruction for DASD (Systems Engineering), participated in JCIDS revisions, developed  guidance for incorporating systems engineering into development contracts, streamlined  the Systems Engineering Plan and Program Protection Plan, and released an update to  the Defense Acquisition Guidebook, chapter 4. Continue to refine policies and  guidance as necessary.
Completing on annual basis: Approved 52 Systems Engineering Plans since 2009,  including 15 in fiscal year 2011.
Completing on annual basis: Review portfolio of 234 programs. In fiscal year 2011,  participated in 73 overarching integrated product team meetings, 6 peer reviews of  acquisition contracts, and conducted 15 Program Support Reviews.
Complete: Developed a set of time-based metrics to assess each program’s ability to  execute its system engineering plans and address risks the office had identified in prior  reviews. The metrics measure program cost, schedule, staffing, reliability, availability  and maintainability, software, integration, performance and manufacturing, and are to be  incorporated into each program’s systems engineering plan and evaluated at various  points in the development process.
Status  Complete: Updated guide for incorporating test and evaluation into acquisition contracts.  Championed updates to DOD Instruction assigning responsibilities and authorities to  Developmental Test and Evaluation office, which is in the process of being updated.  Updated guidance to include reliability factors in the Test and Evaluation Master Plan.  Continue to refine policies and guidance as necessary.
Completing on annual basis: Reviewed and approved 186 Test and Evaluation Master  Plans since 2009, including 44 in fiscal year 2011.
Monitor programs / conduct assessments  Completing on annual basis: Review portfolio of nearly 250 programs. In fiscal year 2011,  participated in 22 defense acquisition board meetings and 59 overarching integrated  product team meetings. The office has also conducted 16 Assessment of Operational  Test Readiness reviews since 2009.
In process: Piloted performance measures on two programs. The measures were then  updated and are being applied to over 40 programs that were selected for reporting in the  fiscal year 2012 joint annual report. The assessments are being used to support the write- up of the program engagement section.
Status  In process: Issued first policy document in May 2012, which is the basis for additional  policy documents. Updating its Operating and Support Cost Estimating Guidebook, which  will address the Reform Act requirement for DOD to issue guidance related to full  consideration of life cycle management and sustainability costs in major defense  acquisition programs.
Not applicable: The Reform Act does not require that the office approve acquisition  documents.
Monitor programs / conduct assessments  Completing on annual basis: Conducted independent cost assessments for Milestone A  and B certification on 30 future and current major defense acquisition programs, since  2009, including 8 in fiscal year 2011. The office conducted 3 Milestone C and 3 Nunn- McCurdy certification reviews in fiscal year 2011.
Not applicable: The Reform Act does not require that the office develop performance  measures.
Status  In process: Developing guidance to assist offices in conducting root cause analyses. The  guidance for conducting performance assessments is expected to be released in early  fiscal year 2013.
Not applicable: The Reform Act does not require that the office approve acquisition  documents.
Monitor programs / conduct assessments  Completing on annual basis: Completed 14 Root Cause Analyses for programs which  have undergone a Nunn-McCurdy breach or were requested by OSD and has completed  26 semi-annual follow-up reports on these programs. Providing OSD with the execution  status of DOD’s portfolio of acquisition programs through the Defense Acquisition  Executive Summary process.
In process: Utilize Defense Acquisition Executive Summary information to identify cost  performance, schedule, funding, and technical performance issues on Major Defense  Acquisition Programs. Continuing to develop performance measures.
section_title: Appendix III: Comments from the Department of Defense