Amphibious Combat Vehicle: Some Acquisition Activities Demonstrate Best Practices; Attainment of Amphibious Capability to be Determined
Released: 2015-10-28T00:00:00

Letter

Background

The AAV is a tracked (non-wheeled) vehicle with the capability to self- deploy—or launch from ships (see figure 1). The AAV has a water speed of approximately six knots, and is usually deployed from within sight of the shore, a factor that poses survivability risks in certain threat environments. According to USMC officials, the AAV has become increasingly difficult to maintain and sustain. As weapons technology and the nature of threats have evolved over the past four decades, the AAV is viewed as having limitations in water speed, land mobility, lethality, protection, and network capability. According to DOD, the need to modernize USMC’s ability to move personnel and equipment from ship to shore is essential. In the last 15 years, USMC has undertaken a number of efforts to do this.

EFV: USMC began development of the EFV in 2000. The EFV was to travel at higher water speeds—around 20 knots—which would have allowed transporting ships to launch the EFV further from shore than the AAVs it was to replace. However, following a 2007 breach of a statutory cost threshold, that program was restructured and subsequently, in 2011, canceled by DOD due to affordability concerns.

ACV: In 2011, the USMC completed initial acquisition documentation providing the performance requirements for a new replacement amphibious vehicle called the ACV. The ACV was expected to be self- deploying with a water speed of 8 to 12 knots which would permit deployment beyond the visual range of the shore, but would not achieve high water speed. It was also expected to provide for sustained operations on shore with improved troop protection. However, USMC leadership then requested an affordability analysis be completed that would explore the technical feasibility of integrating high water speed into ACV development. According to DOD officials, the analysis indicated that achieving high water speed was technically possible but required unacceptable tradeoffs as the program attempted to balance vehicle weight, capabilities, and cost. Meanwhile, the USMC retained a requirement to provide protected land mobility in response to the threat of improvised explosive devices—a requirement the AAV could not meet due to its underbody design. In 2014 we reported that, according to program officials, the program office was in the process of revising its ACV acquisition approach based on this affordability analysis.

ACV 1.1, 1.2 and 2.0: In 2014, the USMC revised its ACV acquisition approach, adopting a plan to develop the ACV in three increments:

The first increment of ACV development—ACV 1.1—is planned to be a wheeled vehicle that would provide improved protected land mobility and limited amphibious capability. The ACV 1.1 is expected to be part of an amphibious assault through the use of surface connector craft to travel from ship to shore. Surface connectors are vessels that enable the transportation of military assets, including personnel, material, and equipment, from a sea base or ship to the shore. ACV 1.1, a successor to the previously suspended Marine Personnel Carrier program, is using prototypes, demonstration testing, and other study results from that program. DOD officials estimated that, in comparing the past Marine Personnel Carrier program and the ACV 1.1 as currently envisioned, the two are about 98 percent the same. Troop capacity—nine for the Marine Personnel Carrier and a threshold, or minimum, of 10 for the ACV 1.1—is the main difference between the two. Figure 2 provides a notional drawing of the ACV 1.1.

The second increment—ACV 1.2—adds two variants of the vehicle for other uses and aims to improve amphibious capability. Program officials anticipate that it will demonstrate amphibious capability that matches the AAV, including the ability to self-deploy and swim to shore. According to DOD officials, ACV 1.2 will be based on the results of ACV 1.1 testing and it is anticipated that some 1.1s will be upgraded with ACV 1.2 modifications.

The third effort, referred to as ACV 2.0, focuses on technology exploration to attain high water speed—a critical capability, according to DOD officials. These technology exploration efforts are seeking design options that may enable high water speed capability without accruing unacceptable trade-offs in other capabilities, cost or schedule. According to officials, ACV 2.0 is a conceptual placeholder for a future decision point when the Marine Corps plans to determine how to replace the AAV fleet, which is expected to occur in the mid- 2020s. High water speed capability may ultimately be achieved through an amphibious vehicle or a surface connector craft.

Knowledge-Based Acquisition Framework

Our prior work on best practices has found that successful programs take steps to gather knowledge that confirms that their technologies are mature, their designs are stable, and their production processes are in control. The knowledge-based acquisition framework involves achieving the right knowledge at the right time, enabling leadership to make informed decisions about when and how best to move into various acquisition phases. Successful product developers ensure a high level of knowledge is achieved at key junctures in development, characterized as knowledge points. Knowledge Point 1 falls early in the acquisition process and coincides with a program’s acquisition’s decision to begin development, referred to as Milestone B. At this knowledge point, best practices are to ensure a match between resources and requirements. Achieving a high level of technology maturity and preliminary system design backed by robust systems engineering is an important indicator of whether this match has been made. This means that the technologies needed to meet essential product requirements have been demonstrated to work in their intended environment. In addition, the developer has completed a preliminary design of the product that shows the design is feasible. Figure 3 identifies the ACV 1.1 acquisition’s status within the DOD acquisition process.

Early ACV Acquisition Activities Demonstrate Best Practices

Our review of the available documents that have been prepared to inform the November 2015 decision to begin system development of ACV 1.1— including the acquisition strategy and an updated 2014 AOA—found that most of the ACV program’s acquisition activities to date reflect the use of best practices. The incremental approach to achieving full capability itself is consistent with best practices. The ACV 1.1 acquisition strategy minimizes program risk by using mature technology, competition, and fixed-price type-contracts when possible. In addition, our analysis of the 2014 AOA found that overall it met best practices. Going forward, however, some elements of the acquisition approach, for example, the program’s plan to hold a preliminary design review (PDR)—a technical review assessing the system design—after beginning development, do not align with best practices and could increase program risk. While some aspects of this acquisition do suggest lower levels of risk, these deviations could potentially increase program risk. GAO will continue to monitor this risk as the program moves forward.

Aspects of the ACV 1.1 Acquisition Strategy Demonstrate Efforts to Minimize Risk

The ACV 1.1 acquisition strategy prepared to inform the upcoming start of engineering and manufacturing development minimizes program risk by following best practices, such as using mature technology, competition, and fixed-price-type contracts when possible.

Technology maturity. The ACV program plans to utilize mature technology in ACV 1.1 development. According to acquisition best practices, demonstrating a high level of maturity before allowing new technologies into product development programs puts programs in a better position to succeed. To support a decision to begin development, a technology readiness assessment (TRA) was performed to assess the maturity of critical technologies to be integrated into the program. DOD defines critical technology elements as new or novel technology that a platform or system depends on to achieve successful development or production or to successfully meet a system operational threshold requirement. In a TRA, identified critical technologies are assessed against a technological readiness level (TRL) scale of 1 to 9. Specifically, a rating of TRL 1 demonstrates “basic principles observed and reported,” and TRL 9 demonstrates “actual system proven through successful mission operations.” Overall, the completed ACV 1.1 TRA assessed the program at TRL 7, indicating demonstration in an operational environment. This assessment was based on the non-developmental nature of the vehicles, the use of mature technology for modifications, and tests and demonstrations of prototype vehicles done under the Marine Personnel Carrier program.

Demonstration in a relevant environment is TRL 6. Demonstration in an operational environment is TRL 7. identified adapting the Remote Weapon Station to the marine environment as a principal program risk because using the system under different operational conditions may have a significant impact on system reliability. While the program has identified additional risk mitigation strategies—including planned component testing during development and development of preventative maintenance procedures—this technology could entail a somewhat higher level of risk than the TRL level suggests and may require additional attention as development begins.

Competition. According to our prior work, competition is a critical tool for achieving the best return on the government’s investment. The ACV acquisition approach has fostered competition in the acquisition process, both through competitive prototyping that took place prior to the start of development and with competition that continues through development until production. Specifically, before the Marine Personnel Carrier program was suspended, the government awarded a contract to test critical sub-systems including the engine, transmission, suspension and hydraulic hardware systems. The government also awarded four contracts for system-level prototypes demonstrating the swim capability, personnel carry capability, and survivability of each company’s vehicle. The Under Secretary of Defense for Acquisition, Technology, and Logistics—the ACV Milestone Decision Authority—has certified to the congressional defense committees that the ACV program had met the competitive prototyping requirement based on the work done under the Marine Personnel Carrier program. In addition, after development begins, the program plans to award ACV 1.1 development contracts to two vendors, maintaining competition until they select one vendor at the start of production.

Contract strategy. When development begins, the ACV program plans to award hybrid contracts to each of the to-be-selected developers. According to program plans, each contract is to utilize three different pricing structures for different activities: fixed-price-incentive for ACV 1.1 vehicle development, firm-fixed-price for the delivery incentive to deliver test vehicles early, and cost-plus-fixed-fee for test support and advanced capability improvements and studies. According to the Federal Acquisition Regulation, it is usually to the Government’s advantage for the contractor to assume substantial cost responsibility and an appropriate share of the cost risk; therefore, fixed-price incentive contracts are preferred when contract costs and performance requirements are reasonably certain. Manufacturing the development vehicles is the largest anticipated portion of ACV development contract costs. According to the ACV 1.1 acquisition strategy, a fixed-price-incentive contract is considered the most appropriate contract type to utilize for the vehicle’s development because the vehicles themselves are non-developmental in nature but there is some risk related to the integration of selected systems, such as the Remote Weapon Station, and other modifications required to meet USMC requirements. Meanwhile, the strategy states that the delivery incentive is to be a firm-fixed-price, as the fee is a set dollar amount based on how early the vehicles are delivered and is not subject to adjustment based on the vendor’s costs.

Under cost-reimbursement contract types, such as a cost-plus-fixed-fee contract, the government bears the risk of increases in the cost of performance. Cost-reimbursement contract types are suitable when uncertainties in requirements or contract performance do not permit the use of fixed-price contract types. A cost-plus-fixed-fee structure is planned for test support before and after the start of production, vehicle transportation and other test-related activities. According to program officials, the scope and nature of these activities are difficult to predict, making the cost-plus-fixed-fee structure appropriate. Officials also stated that the cost-plus-fixed-fee activities are expected to comprise about 11 percent of the total contract value.

Requirements and cost estimates. Additional key documents have been prepared, or are underway, in accordance with DOD policy. The ACV 1.1 Capabilities Development Document, providing the set of requirements for development, is tailored specifically for ACV 1.1. In accordance with DOD policy, the ACV 1.1 Capabilities Development Document was validated prior to the release of the ACV 1.1 request for proposal in March 2015. In addition, best practices and DOD policy also call for the development of an independent cost estimate prior to the start of development. According to agency officials, the independent cost estimate is underway and will be prepared for the Milestone B decision. The acquisition strategy identifies no funding shortfalls for the program as of the fiscal year 2016 President’s budget submission.

2014 ACV Analysis of Alternatives Met Best Practices

Our assessment of the 2014 AOA found that overall it met best practices for AOAs and is, therefore, considered reliable. An AOA is a key first step in the acquisition process intended to assess alternative solutions for addressing a validated need. AOAs are done or updated to support key acquisition decision points. The USMC completed an AOA update for ACV 1.1 in late 2014 to support the release of the ACV 1.1 request for proposal. Over the years, other AOAs have been completed for related acquisitions, including the EFV, the Marine Personnel Carrier and the previous version of the ACV considered in 2012. These previous AOAs and other supporting studies comprise a body of work that has informed the most recent ACV AOA update as well as the ACV 1.1 acquisition as a whole.

AOAs can vary in quality, which can affect how they help position a program for success. We have previously identified best practices for the development of AOAs. Considered in the context of the related AOA body of work, the ACV AOA met 15 of the 22 AOA best practices, including ensuring that the AOA process was impartial and developing an AOA process plan, among others. Further, four of the remaining best practices were substantially met, two were partially met, and one was minimally met. For example, best practices call for the documentation of all assumptions and constraints used in the analysis. We found that the 2014 AOA does not include a full list of assumptions and constraints and any assumptions or constraints from previous analysis, if relevant, were not updated or referenced in the new analysis. As a result, it could be difficult for decision makers to make comparisons and trade-offs between alternatives. Appendices I and II provide more information on the methodology used in this analysis and appendix III provides the results of our AOA analysis in greater detail. DOD’s Cost Assessment and Program Evaluation staff also reviewed the 2014 AOA and found that it was sufficient. However, they identified a few areas of caution, including recommending additional testing of land mobility to further verify USMC assertions that the wheeled ACV 1.1 would have the same mobility in soft soil as tracked vehicles.

Accelerated Schedule Presents Program Risks

According to USMC officials, the ACV program is pursuing an aggressive schedule in order to obtain ACV 1.1 initial operational capability in fiscal year 2020. The program is scheduled to hold its PDR after development starts, a deviation from best practices. In addition, according to program officials, as a result of the aggressive acquisition schedule, the program plans on a higher level of concurrency between development testing and production than would take place under a more typical acquisition schedule. This aggressive schedule may likely have congressional decision makers approve funds to begin production based on little to no evidence from the testing of delivered ACV 1.1 prototypes. Some factors may mitigate the risk posed by this acceleration, for example, program officials have stated that all required testing will take place prior to the start of production. However, further attention may be warranted in our future reviews of the program’s schedule.

The ACV 1.1 program is planning to hold its PDR about 90 days after development begins and to combine its PDR and the critical design review (CDR) into one event. Best practices recommend that the PDR is held before development begins in order to increase the knowledge available to the agency when development starts, for example, increasing confidence that the design will meet the requirements established in the Capabilities Development Document. The absence of a PDR introduces some risk by postponing the attainment of knowledge until after development begins and reducing scheduled time to address any design issues that may arise. In addition, it is a best practice to demonstrate design stability at the system-level CDR, completing at least 90 percent of engineering drawings at that time. Combining the PDR and CDR may limit the time available to the program to address any issues identified and ensure that sufficient knowledge is attained prior to the program moving forward. For example, in a 2006 report, we found that the EFV program’s CDR was held almost immediately after the start of development—similar to the approach for ACV 1.1—and before the system integration work had been completed. Testing of the early prototypes continued for three years into system development, well after the tests could inform the CDR decision. Best practices call for system integration work to be conducted before the CDR is held. According to DOD officials, the ACV 1.1 PDR will be held after Milestone B because contracts are not planned to be awarded prior to that time. In addition, DOD officials stated that the technological maturity of ACV 1.1 reduces risk and permits both the waiver of the PDR requirement and the consolidation of the reviews. While the use of mature technology could suggest a reduced risk from this deferral, we believe that contracts could have been awarded earlier in the acquisition process in order to facilitate a PDR prior to development start.

The current ACV 1.1 program schedule demonstrates concurrency between testing and production that could represent increased program risk. According to agency officials, approximately one year of development testing will take place prior to the program’s production decision in order to assess production readiness. Another ten months of testing will continue after the start of production. The intent of developmental testing is to demonstrate the maturity of a design and to discover and fix design and performance problems before a system enters production. According to agency officials, the adoption of an accelerated fielding schedule is behind the level of overlap between developmental testing and production. They stated that they plan to have completed all development testing and operational assessment required to support the production decision by the time that decision is made. DOD policy allows some degree of concurrency between initial production and developmental testing and, according to our prior work, some concurrency may be necessary when rapidly fielding urgently needed warfighter capabilities. However, our past work has also shown that beginning production before demonstrating that a design is mature and that a system will work as intended increases the risk of discovering deficiencies during production that could require substantial design changes and costly modifications to systems already built. A detailed test plan will not become available until Milestone B as is expected for acquisition programs. When such a plan is available, we will further assess the risk presented by this approach.

Moreover, under the current ACV 1.1 program schedule, Congress may likely be called upon to provide production funding for ACV 1.1 production based on little to no evidence from the testing of delivered ACV 1.1 prototypes. The program is scheduled to make a production decision, and select one vendor, in fiscal year 2018. Under the normal budget process, Congress would be provided the request for funding that production with the President’s budget in February 2017, around the same time that the prototype ACV 1.1 vehicles are scheduled to be delivered. In the event that the development testing schedule experiences delays and key tests are postponed until after the planned production decision, the program may face increased risk.

Incremental Development of Future Amphibious Capability Is Key to Current ACV Strategy

The success of the ACV acquisition strategy depends upon the attainment of improved amphibious capabilities over time. The first increment, ACV 1.1, is not expected to have ship to shore amphibious capability and thus is planned to use Navy surface connectors to travel from ship to shore. The USMC and the Navy have coordinated the planned operation of ACV 1.1 with surface connectors to ensure compatibility and availability. The ACV acquisition intends to rely heavily upon realizing a fully amphibious ACV 1.2, providing AAV-equivalent water mobility and the ability to self-deploy. However, the exact nature of ACV 1.2 and 2.0 is unknown at this time. Achieving the planned capabilities of future ACV increments is highly dependent upon ACV 1.1 attaining its planned amphibious capability.

ACV 1.1 Expects to Rely on Surface Connector Craft for All Ship to Shore Transportation

While ACV 1.1 is expected to have shore to shore amphibious capability, which would enable the vehicle to cross rivers and inland waterways, the vehicle is also expected to rely on Navy surface connector craft for ship to shore transportation. Connectors have become increasingly important as USMC vehicles have grown in weight. According to USMC analysis, about 86 percent of USMC expeditionary force assets are too heavy or over-sized for air transport, and need to be transported by surface connectors.

The ACV 1.1 requirements include transportability by currently available and planned Navy surface connectors. Because several surface connectors can transport the ACV 1.1, the selection of specific surface connectors is planned to be based on an evaluation of mission needs and connector capabilities. Some current and planned Navy surface connectors that could transport ACV 1.1 are described below. Appendix IV provides additional information on the key capabilities of these connectors.

Landing Craft Air Cushion (LCAC). The LCAC is a high speed hovercraft that supports rapid movement from ship to shore, such as during an amphibious assault. The LCAC is one of the primary connectors that provide ship to shore transportation of equipment, personnel, and vehicles. The LCAC, which can access about 70 percent of the world’s beaches, is optimized towards major combat operations and forcible entry. The Navy currently has a fleet of 72 LCACs which have received upgrades as a result of a service life extension program effort. The Navy also plans to provide additional LCAC maintenance until replacement craft are acquired.

Ship to Shore Connector (SSC). The Navy plans to replace each LCAC with an SSC. The SSC, similar in design to the LCAC, is planned to maintain or improve upon LCAC capabilities with an increased payload capacity, a longer service life, and the ability to operate in more harsh marine environments. SSC is planned to reach initial operational capability of 6 craft in 2020 and full operational capability in 2027.

Landing Craft Utility (LCU). The LCU is a utility connector that supports ship to shore movement in amphibious assaults and also participates in a variety of other missions. The LCU has a large range and payload capacity, but operates at a slower speed compared to the LCAC. According to Navy officials, the LCU can access about 17 percent of the world’s beaches, and stops at the waters’ edge in order to unload its cargo.

Surface Connector (X) Replacement (SC(X)R). According to Navy officials, the aging LCU craft are planned to be replaced by SC(X)R craft in order to maintain a total of 32 LCUs and SC(X)Rs. According to the Surface Connector Council, the SC(X)R is likely to be larger and show improvements in materials, propulsion, maintainability, and habitability. Production for the SC(X)R is planned to begin in 2018.

Expeditionary Fast Transport (EPF). The EPF, formerly known as the Joint High Speed Vessel, is a commercial-based catamaran that provides heavy-lift, high-speed sealift mobility. The EPF uses a ramp system to allow vehicles to off-load at shipping ports or where developed infrastructure is unavailable (referred to as austere ports). The EPF is planned to reach full operational capability in the year 2019.

Figure 4 illustrates three examples of how various surface connectors could be used to transport ACV 1.1 from ship to shore. For example, ACVs could be loaded onto an Expeditionary Transfer Dock (ESD) and then on to LCACs or SSCs while the ESD maneuvers towards the shore. The LCACs or SSCs would then launch from the ESD and transport the ACVs to shore. The ACV could also be off-loaded at an advanced base —such as an island located within the operational area—and then loaded onto a EPF for transport to a developed or austere port. Finally, the ACVs could be directly loaded from ships on to a LCU or SC(X)R and taken to shore. This graphic includes selected examples only, and does not represent all possible transportation options.

Navy Officials Have Taken Actions to Mitigate Potential SSC Risks

SSC acquisition risks may have consequences for employment of ACV 1.1. The Navy has identified that it requires a combined fleet of at least 72 operational LCACs and SSCs to support ship to shore transportation demands. However, the Navy previously anticipated a lack of available connectors from the year 2015 through 2024, with a maximum ‘gap,’ or shortage, of 15 craft in 2019. Navy officials said that this ‘connector gap’ has been mitigated with the extension of the LCAC service life extension program and acceleration of the SSC acquisition.

In a previous assessment of the SSC program, we found that the Navy recognizes three SSC technologies as potential risk areas, for which the Navy recommended further testing. According to officials, since that report, the Navy has completed additional testing for software, drivetrain components, and engine endurance to further develop and reduce the risk of these technologies. Navy officials said the SSC program plans to continue testing these technologies and remains on-schedule. However, the SSC program entered production in 2015, more than 2 years before the estimated delivery of the test vehicle. This concurrency of development and production creates a potential risk of schedule overruns if deficiencies in the design are not discovered until late in testing and retrofits are required for previously produced craft. Navy officials said that the LCAC service life could be further extended with additional sustainment funding in the event of SSC acquisition delays.

USMC and Navy Coordinated ACV and Surface Connector Plans with Various Mechanisms

The USMC and Navy regularly coordinate on the ACV 1.1 to facilitate the future use of the surface connector fleet through the Joint Capabilities Integration Development System (JCIDS), the Surface Connector Council, and other communication.

JCIDS. The JCIDS process is a DOD-wide process to identify and assess capability needs and their associated performance criteria. The Capabilities Development Document for the ACV 1.1 was developed as part of the JCIDS process. The document, among other things, identified key systems attributes, key performance parameters, and design requirements for the ACV 1.1 with input from the USMC, the Navy, and others. For example, it included design requirements that allow the SSC to transport two ACVs, and ensure that ACVs can be transported by other connector craft as well.

Surface Connector Council and working group. The Surface Connector Council serves as a mechanism through which the USMC and Navy coordinate activities related to surface connectors that are used for amphibious shipping. The council has two co-chairs: the Director of the Navy’s Expeditionary Warfare Division and the Director of the USMC Capabilities Development Directorate who is also the Deputy Commandant for Combat Development and Integration. The council membership is drawn from several offices from both the Navy and the USMC. The Council is required to meet at least biannually but, according to Navy officials, in practice the Council generally meets quarterly. At these meetings, the Council has previously discussed ACV program risks, such as connector availability and the scarcity of space on connectors, and associated risk mitigation strategies, according to Navy officials. The Surface Connector Council also has a working level forum, known as the Surface Connector Roundtable, which meets on a monthly basis according to Navy officials.

Informal discussions. In addition to coordination through JCIDS and the Surface Connector Council, officials said that informal discussions between USMC and Navy officials occur frequently to coordinate the ACV and connector programs.

Future Amphibious Capability to be Determined

The exact nature of the ACV’s future amphibious capability is not yet known. USMC officials are confident that the ACV 1.1 would not only meet its minimum requirements for shore to shore swim capability, but may exceed those requirements and be able to swim from ship to shore. Based on tests and demonstrations to date, program officials also expressed confidence that ACV 1.2 will build on the ACV 1.1 capabilities and have the ability to self-deploy from ships. However, according to DOD officials, the capabilities of the ACV 1.2 are dependent upon the success of ACV 1.1 development. If the ACV 1.1 does not demonstrate the expected amphibious capabilities, then more development than currently anticipated may be required for ACV 1.2 to achieve ship to shore amphibious capability and greater effort may be needed to retro-fit ACV 1.1 vehicles to achieve the same capabilities. However, if ACV 1.1 demonstrates greater than expected amphibious capability, then the progression towards achieving the plans for the ACV 1.2 may be easier. Program documentation and analysis to date have been done to develop the ACV 1.1 strategy and plans and to support ACV 1.1 decisions. According to DOD officials, the USMC has not yet determined whether the development of ACV 1.2 will be done through improvements within the same program or as a separate program from ACV 1.1. DOD officials stated that the development of ACV 1.1 and 1.2 amphibious capabilities is also expected to impact the nature of ACV 2.0. According to DOD officials, with the ACV 2.0 decision, the ACV program expects to achieve high water speed, a long-standing goal and a significant increase from the current amphibious goals identified for ACV 1.1.

The current USMC amphibious strategy plans for an evolving mix of ACVs and upgraded and legacy AAVs that are to maintain the needed combination of capabilities at any one time. According to USMC officials, over time, the ACV program plans to replace portions of the AAV fleet with ACV increments as they become available. This USMC strategy, and the analysis that supports it, is based on the assumption that ACV 1.2 will reach a desired level of amphibious capability and that ACV 1.1 vehicles can be upgraded to that level. If, however, those or other key capabilities cannot be achieved, revisiting the USMC’s strategy prior to making production decisions for ACV 1.1, particularly addressing changes to its overall amphibious strategy and potentially updating its analysis of alternatives, will be important. In addition, when and how the USMC will achieve the amphibious capability envisioned for ACV 2.0 remains to be determined, according to DOD officials. We will continue to monitor these issues along with the program’s performance against best practices as it progresses toward the Milestone C production decision currently planned for the second quarter of fiscal year 2018.

Agency Comments and Our Evaluation

We are not making any recommendations in this report. DOD provided written comments on a draft of this report. The comments are reprinted in appendix V.

In commenting on a draft of this report, DOD stated that it believes its efforts on this program are aligned with our best practices and that our report appears to underestimate ACV 1.1’s planned technical maturity and associated risks. DOD stated that the vehicle is beyond the traditional PDR and CDR level of maturity and conducting a combined PDR and CDR is appropriate for the level of risk identified by the Program Manager. As we stated in this report, the program’s plan to hold a PDR after beginning development does not align with best practices and combining the PDR and CDR may limit the time available to the program to address any issues identified and ensure that sufficient knowledge is attained prior to the program moving forward. Further, as we stated earlier, while some aspects of this acquisition do suggest lower levels of risk, these deviations could potentially increase program risk—risks that we will continue to monitor as the program moves forward.

DOD also provided technical comments that were incorporated, where appropriate.

We are sending copies of this report to interested congressional committees; the Secretary of Defense; the Under Secretary of Defense for Acquisition, Technology, and Logistics; the Secretary of the Navy; and the Commandant of the Marine Corps. This report also is available at no charge on GAO’s website at http://www.gao.gov.

Should you or your staff have any questions on the matters covered in this report, please contact me at (202) 512-4841 or makm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VI.

Appendix I: Best Practices for the Analysis of Alternatives Process

Background and Introduction

Many guides have described an approach to analyses of alternatives (AOAs); however, there is no single set of practices for the AOA process that has been broadly recognized by both the government and private- sector entities. GAO has identified 22 best practices for an AOA process by (1) compiling and reviewing commonly mentioned AOA policies and guidance used by different government and private-sector entities and (2) incorporating experts’ comments on a draft set of practices to develop a final set of practices.

These practices can be applied to a wide range of activities in which an alternative must be selected from a set of possible options, as well as to a broad range of capability areas, projects, and programs. These practices can provide a framework to help ensure that entities consistently and reliably select the project alternative that best meets mission needs. The guidance below is meant as an overview of the key principles that lead to a successful AOA process and not as a “how to” guide with detailed instructions for each best practice identified.

The 22 best practices that GAO identified are grouped into the following five phases: 1. Initialize the AOA process: includes best practices that are applied before starting the process of identifying, analyzing, and selecting alternatives. This includes determining the mission need and functional requirements, developing the study time frame, creating a study plan, and determining who conducts the analysis. 2. Identify alternatives: includes best practices that help ensure the alternatives to be analyzed are sufficient, diverse, and viable. 3. Analyze alternatives: includes best practices that compare the alternatives to be analyzed. The best practices in this category help ensure that the team conducting the analysis uses a standard, quantitative process to assess the alternatives. 4. Document and review the AOA process: includes best practices that would be applied throughout the AOA process, such as documenting all steps taken to initialize, identify, and analyze alternatives and to select a preferred alternative in a single document. 5. Select a preferred alternative: includes a best practice that is applied by the decision maker to compare alternatives and to select a preferred alternative.

The five phases address different themes of analysis necessary to complete the AOA process and comprise the beginning of the AOA process (defining the mission needs and functional requirements) through the final step of the AOA process (select a preferred alternative).

There are three key entities that are involved in the AOA process: the customer, the decision maker, and the AOA team. The customer refers to the program office, service, or agency that identifies a mission need (e.g. a credible gap between current capabilities and those required to meet the goals articulated in the strategic plan). The decision maker is the person or entity that signs off on the final decision and analysis documented by the AOA report. The decision maker refers to the program manager (or alternate authority figure identified early in the AOA process) who will select the preferred alternative based on the established selection criteria. The AOA team is the group of subject matter experts who are involved in the day-to-day work of the AOA process and work to develop the analysis that is the foundation of the AOA process.

Conforming to the 22 best practices helps ensure that the preferred alternative selected is the one that best meets the agency’s mission needs. Not conforming to the best practices may lead to an unreliable AOA, and the customer will not have assurance that the preferred alternative best meets the mission needs. Table 1 shows the 22 best practices and the five phases.

Some best practices included in a phase can take place concurrently and do not have to follow the order presented in table 1. The phases should occur in sequence to prevent bias from entering the analysis and adding risk that the AOA team will analyze alternatives that have not been defined. However, the document and review phase can be done at any stage throughout the AOA process. For example, best practice 5 (define selection criteria) can be done at the same time as best practice 6 (weight selection criteria). On the other hand, best practice 20 (ensure AOA process is impartial) can be done at the end of every step or every phase to ensure the impartiality of the AOA as it progresses. The best practices represent an overall process that results in a reliable AOA that can be easily and clearly traced, replicated, and updated. Figure 5 shows the AOA process and how the steps in each phase are interrelated.

Review the AOA Process

An important best practice is an independent review of the AOA process. It is important that the AOA process and its results be validated by an organization independent of the program office and the project’s chain of command, to ensure that a high-quality AOA is developed, presented, and defended to management. This process verifies that the AOA adequately reflects the program’s mission needs and provides a reasonable assessment of the cost and benefits associated with the alternatives.

One reason to independently validate the AOA process is that independent reviewers typically rely less on assumptions alone and, therefore, tend to provide more realistic analyses. Moreover, independent reviewers are less likely to automatically accept unproven assumptions associated with anticipated savings. That is, they bring more objectivity to their analyses, resulting in a reality check of the AOA process that reduces the odds that management will invest in an unreasonable alternative.

To that end, we established four characteristics that identify a high- quality, reliable AOA process. These characteristics would evaluate if the AOA process is well-documented, comprehensive, unbiased, and credible. “Well-documented” means that the AOA process is thoroughly described in a single document, including all source data, clearly detailed methodologies, calculations and results, and that selection criteria are explained. “Comprehensive” means that the AOA process ensures that the mission need is defined in a way to allow for a robust set of alternatives, that no alternatives are omitted and that each alternative is examined thoroughly for the project’s entire life-cycle. “Unbiased” means that the AOA process does not have a predisposition toward one alternative over another; it is based on traceable and verifiable information. “Credible” means that the AOA process thoroughly discusses the limitations of the analyses resulting from the uncertainty that surrounds both the data and the assumptions for each alternative.

Table 2 shows the four characteristics and their relevant AOA best practices.

Appendix II: Scope and Methodology

To determine how the ACV program’s efforts compare with best practices, we reviewed program documentation and other materials for the ACV acquisition, including the acquisition strategy, technology readiness assessment, and the Capabilities Development Document. We identified acquisition best practices based on our extensive body of work in that area and Department of Defense (DOD) guidance, and used this information to analyze the proposed ACV acquisition approach and acquisition activities to date. We also reviewed our previous work on the ACV and EFV programs. In addition, we interviewed program and agency officials from the USMC’s Advanced Amphibious Assault program office and Combat Development and Integration, Analysis Directorate, the Office of the Assistant Secretary of the Navy for Research, Development, and Acquisition, and the Office of the Secretary of Defense, Cost Assessment and Program Evaluation.

To determine the extent to which the 2014 ACV Analysis of Alternatives (AOA) demonstrated the use of best practices, we worked with USMC officials to identify the body of analyses that informed the 2014 AOA. Different pieces of each report or analysis in the full body of work were relevant to different best practices. Because the 2014 ACV AOA is part of a larger body of related work that informs this analysis, we then worked with GAO specialists to discuss the 22 AOA best practices and categorize each as either “individual” or “combined.” Best practices labeled “individual” have been assessed based on only the 2014 ACV Analysis of Alternatives final report. Best practices noted as “combined” were assessed referring to the full body of work that, according to USMC officials, has informed the analysis of alternatives process. We then compared the 22 best practices to the 2014 AOA or the full body of AOA analysis, as determined above. We used a five-point scoring system to determine the extent to which the AOA conforms to best practices. To score each AOA process, (1) two GAO analysts separately examined the AOA documentation received from the agency and then agreed on a score for each of the 22 best practices, then (2) a GAO AOA specialist independent of the engagement team reviewed the AOA documentation and the scores assigned by the analysts for accuracy and cross-checked the scores in all the analyses for consistency. We first used this scoring system to determine how well the AOA conformed to each best practice. We then used the average of the scores for the best practices in each of four characteristics—well-documented, comprehensive, unbiased, and credible—to determine an overall score for each characteristic. We sent our draft analysis to DOD for review. They provided technical comments and additional documentation that we incorporated to ensure our analysis included all available information. We then used the same methodology and scoring process explained above to revise the analysis based on their technical comments and any additional evidence received. If the average score for each characteristic was “met” or “substantially met,” we concluded that the AOA process conformed to best practices and therefore could be considered reliable.

To determine how the increments of ACV are to achieve amphibious capability, we reviewed program documentation from the ACV acquisition, including the acquisition strategy and the Concept of Employment, as well as program documentation for Navy surface connector programs, including the Ship to Shore Connector Capabilities Development Document and the Surface Connector Council charter. We also interviewed USMC officials from the Combat Development and Integration, Capabilities Development Directorate and Seabasing Integration Division, as well as U.S. Navy officials from the Naval Sea Systems Command.

To update and refine the AOA best practices identified in prior GAO work, we solicited comments from a set of over 900 internal and external experts on how to improve the previous set of best practices. All comments and changes were vetted during three vetting sessions with internal GAO experts. The resulting changes include the consolidation of some best practices, reducing the number from 24 to 22, and the establishment of four characteristics that identify a high-quality, reliable AOA process.

Appendix III: Amphibious Combat Vehicle 2014 Analysis of Alternatives Compared with Best Practices

Overall, the DOD’s ACV analysis of alternatives (AOA) met the best practices we identified. Table 3 below describes our analysis of DOD’s AOA compared with best practices.

Table 4 provides the average score of the best practices under each characteristic. See appendix I for an explanation of how individual best practices are grouped under each characteristic. Because the overall assessment ratings for each of the four characteristics are substantially met or met, we concluded that the AOA process conformed to best practices and can be considered reliable.

Appendix IV: Additional Information on Selected Surface Connectors and Key Capabilities

Current Quantity or Planned Quantity 72

Operating Sea State with Significant Wave Height (SWH) 2 (1.0 SWH) 3 (4.1 SWH)

Appendix V: Comments from the Department of Defense

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

Key contributors to this report were Bruce H. Thomas, Assistant Director; Betsy Gregory-Hosler, analyst-in-charge; Zachary Sivo; Marie Ahearn; Brian Bothwell; Jennifer Echard; Kristine Hassinger; Katherine Lenane; Jennifer Leotta; David Richards; Karen Richey; Robert S. Swierczek; Hai Tran; and Ozzy Trevino.