Defense Acquisitions: The Expeditionary Fighting Vehicle Encountered Difficulties in Design Demonstration and Faces Future Risks
Published: 2006-05-01T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Marine Corps' Expeditionary Fighting Vehicle (EFV) is the Corps' number-one priority ground system acquisition program and accounts for 25.5 percent of the Corps' total acquisition budget for fiscal years 2006 through 2011. It will replace the current amphibious assault craft and is intended to provide significant increases in mobility, lethality, and reliability. We reviewed the program under the Comptroller General's authority to examine (1) the cost, schedule, and performance of the EFV program during system development and demonstration; (2) factors that have contributed to this performance; and (3) future risks the program faces as it approaches production.
section_title: What GAO Found
paragraphs: Although the EFV program had followed a knowledge-based approach early in development, its buying power has eroded during System Development and Demonstration (SDD). Since beginning this final phase of development in December 2000, cost has increased 45 percent. Unit costs have increased from $8.5 million to $12.3 million. The program schedule has grown 35 percent or 4 years, and its reliability requirement has been reduced from 70 hours of continuous operation to 43.5 hours. Program difficulties occurred in part because not enough time was allowed to demonstrate maturity of the EFV design during SDD. The SDD schedule of about 3 years proved too short to conduct all necessary planning and to incorporate the results of tests into design changes, resulting in schedule slippages. In addition, several significant technical problems surfaced, including problems with the hull electronic unit, the bow flap, and the hydraulics. Reliability also remains a challenge. Three areas of significant risk remain for demonstrating design and production maturity that have potential significant cost and schedule consequences. First, EFV plans are to enter low-rate initial production without requiring the contractor to demonstrate that the EFV's manufacturing processes are under control. Second, the EFV program will begin low-rate initial production without the knowledge that software development capabilities are sufficiently mature. Third, two key performance parameters--reliability and interoperability--are not scheduled to be demonstrated until the initial test and evaluation phase in fiscal year 2010--about 4 years after low-rate initial production has begun.

section_title: Letter
paragraphs: Congress continues to express concerns over both the costs and the cost  growth of the Department of Defense’s (DOD) major acquisition programs  and not following its own acquisition policies. In the November 2005,  hearing on DOD Acquisition Reform, the House Armed Services  Committee noted that DOD’s acquisition costs and capabilities were  increasing so much for individual systems that the nation will not be able  to afford enough of them to support its missions; it also observed that the  symptoms of this problem include increasing costs and programs ignoring  internal regulations and processes.
We have reported on widespread and persistent cost, schedule, and  performance problems with major weapon system developments and  DOD’s inability to resolve them. Over the last 9 years, we have  benchmarked successful commercial and defense development programs  and identified the key characteristics for getting better outcomes as being  knowledge-based. Successful programs insist on having key product  knowledge demonstrated at key points in a new development.
We have found that a sound business case at the beginning of the system  development and demonstration (SDD) phase is essential for the  successful completion of a weapon system program. Demonstrated  knowledge at key junctures is at the core of the business case. The basic  elements of a sound business case at the start of SDD include:     A match must be made between the customer’s needs and mature  technology. We refer to this as knowledge point 1.
The acquisition strategy for SDD should provide for demonstrating:     Design stability at the time of the critical design review  (knowledge point 2).
The design meets performance requirements, is reliable, and can  be produced within cost, schedule, and quality targets before  production begins (knowledge point 3).
A realistic cost estimate is made to support the acquisition strategy.    Sufficient funds are available to cover realistic program costs.
In sum, successful programs insist on having key product knowledge  demonstrated at key points in a new development.
Starting in October 2000, DOD incorporated a knowledge-based approach  in its policy that guides major acquisitions and expanded this approach in  its May 2003 policy. The way to implement this policy is through decisions  on individual programs. As we have reported, most individual programs do  not follow a knowledge-based approach, preferring instead to proceed  without adequate knowledge and to accept the consequences of lost  buying power that attend subsequent cost increases.
The Marine Corps’ Expeditionary Fighting Vehicle (EFV) is a major  acquisition program that did show indications of following a knowledge- based approach and other best practices. For example, the program earlier  adopted best practices in its implementation of Integrated Product Teams  and has trained its program office staff on this acquisition improvement  initiative. In addition, as we have reported, the earlier EFV program has  been a leader both in the use of Integrated Product Teams and Cost as an  Independent Variable. The EFV program has since been used by the  Defense Acquisition University as a lessons-learned case study for training  acquisition program managers.
We reviewed the EFV program under the Comptroller General’s authority  to determine how it is performing against its business case. Specifically,  this report addresses:  the cost, schedule, and performance of the EFV program during  SDD;   factors that have contributed to this performance; and   future risk the program faces as it approaches production.
In conducting our review, we used knowledge-based acquisition strategy  principles as a framework. Appendix I contains details of our approach.  We conducted our work from May 2005 to May 2006 in accordance with  generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: The EFV is the Corps’ number-one priority ground system acquisition  program and is the successor to the Marine Corps’ existing amphibious  assault vehicle. It is designed to transport troops from ships offshore to  their inland destinations at higher speeds and from farther distances, and  to be more mobile, lethal, reliable, and effective in all weather conditions.  It will have two variants—a troop carrier for 17 combat-equipped Marines  and a crew of 3 and a command vehicle to manage combat operations in  the field. The Marine Corps’ total EFV program requirement is for   1,025 vehicles. Figure 1 depicts the EFV system.
The EFV’s total acquisition cost is currently estimated to be about   $12.6 billion. In addition, the EFV accounts for a substantial portion of the  Marine Corps’ total acquisition budget for fiscal years 2006 through 2011,  as figure 2 shows.
The EFV program began its program definition and risk reduction phase in  1995, and was originally referred to as the Advanced Assault Amphibious  Vehicle. The Marine Corps’ existing assault amphibious vehicle was  originally fielded in 1972 and will be over 30 years old when the EFV is  fielded. Several Marine Corps studies identified deficiencies in the existing  vehicle, including the lack of necessary lethality to defeat projected  emerging threats. Despite efforts to extend the service life of the existing  vehicle, Marine Corps officials stated that serious warfighting deficiencies  remained. The studies concluded that the existing vehicle was unable to  perform the type of combat missions envisioned by the Marine Corps’  emerging combat doctrine and that a new vehicle was needed.
In September 2003, DOD officially changed the name of the new vehicle to  the EFV, which was in keeping with the Marine Corps’ cultural shift from  the 20th century force defined by amphibious operations to a 21st century  force focusing on a broadened range of employment concepts and  possibilities across a spectrum of conflict. The new vehicle is a self- deploying, high water-speed, amphibious, armored, tracked vehicle, and is  to provide essential command, control, communications, computers, and  intelligence functions for embarked personnel and EFV units. These  functions are to be interoperable with other Marine Corps systems as well  as with Army, Air Force, Navy, and NATO systems. The EFV transitioned  to SDD in December 2000. The use of a knowledge-based acquisition  approach was evident at the onset of the EFV program. Early in the  program at the start of program definition and risk reduction, the Marine  Corps ensured that four of the five critical program technologies were  mature. Although the fifth technology (the moving map navigation  technology, which provides situational awareness) was not mature at this  same time, it was sufficiently matured after the program transitioned to  SDD. Furthermore, the EFV design showed evidence of being stable by the  completion and release of design drawings. At critical design review,   84 percent of the drawings were completed and released. The program  now has 100 percent of the EFV drawings completed. Program officials  expect that only about 12 percent of the design drawings are likely to be  changed in the future as a result of planned reliability testing.
section_title: Cost, Schedule, and Other Problems Have Reduced EFV Buying Power
paragraphs: Since entering SDD in December, 2000, the EFV program’s total cost has  grown by about $3.9 billion, or 45 percent. Production quantities have  been reduced by about 55 percent over fiscal years 2006-2011, thereby  reducing the capabilities provided to the warfighter during this period.  Cost per vehicle has increased from $8.5 million to $12.3 million. However,  total quantities remain unchanged. During the same period, the EFV’s  development schedule has grown by about 4 years, or 35 percent.  Furthermore, a key requirement has been lowered. EFV reliability—a key  performance parameter—has been reduced from 70 hours of continuous  operation to 43.5 hours. Thus, overall EFV buying power has been  reduced, for it will now take substantially more money than was estimated  at the start of SDD to acquire the same number of vehicles later and more  slowly, and with a reduced operational reliability requirement.
subsections: section_title: EFV Costs and Schedule Have Grown Significantly Since Entering SDD
paragraphs: Since entering SDD in December 2000 and holding the SDD critical design  review in January, 2001, the EFV program’s total acquisition cost has  grown by about $3.9 billion, or 45 percent, to $12.6 billion. Figure 3 shows  how costs have grown over time.
While total quantities have not changed, production quantities over fiscal  years 2006-2011 were reduced by about 55 percent, from 461 vehicles to  208. This means that the warfighter will get the capability the EFV  provides more slowly.
The EFV program has been rebaselined three times since SDD began, as  shown in table 1.
Because the rebaselines have occurred incrementally over time, the EFV  program has not previously been required to submit a unit cost increase  report to Congress. Congress in 1982 enacted the unit cost reporting  statute, now codified in 10 USC 2433, which is commonly referred to as  Nunn-McCurdy, after the congressional leaders responsible for the  requirement. The statute required the Secretary of Defense to certify a  program to Congress when the unit cost growth in constant dollars  reaches 25 percent above the most recent rebaseline cost estimate and  report to Congress when it reaches 15 percent. The National Defense  Authorization Act for fiscal year 2006 made changes to Nunn-McCurdy.  The primary change that affects the EFV program was the additional  requirement to report 30 percent unit cost growth above the original  baseline estimate approved at SDD. The EFV program recently reported an  increase in the EFV’s program average unit cost increase of at least   30 percent above its original baseline estimate at SDD. Although the EFV  program acquisition unit costs have increased by about at least 30 percent  since SDD began, no single increase between rebaselines has reached the  15 percent reporting threshold.
Overall, the program schedule has grown by 48 months or 35 percent from  December 2000 at the start of SDD to the most recent rebaselining in  March 2005. This schedule growth has delayed the occurrence of key  events. For example, the EFV program was originally scheduled to provide  the Marine Corps with its initial operational capability vehicles in  September 2006, but is now scheduled to provide this capability in  September 2010. Details of key event schedule changes are shown in   table 2.
section_title: Reliability Requirement Reduced
paragraphs: In 2005, the Marine Corps received approval to lower the EFV’s reliability  requirement from 70 hours before maintenance is needed to 43.5 hours  before maintenance is needed. This decision was based on a revised  analysis of the EFV’s mission profile and the vehicle’s demonstrated  reliability. At the start of SDD, the EFV’s operational reliability  requirement was 70 hours of operation before maintenance is needed.  Program officials told us this 70-hour requirement was based on the EFV’s  mission profile at the time, which called for a “do-all” mission for one   24.3 hour period of operation. The original reliability growth plan  anticipated that this requirement would be met after initial operational test  and evaluation, which was then planned for August 2007.
In 2002, the Marine Corps’ Combat Development Command performed an  independent analysis of the original 70-hour reliability requirement and  determined that it was likely that it would be very difficult to achieve.  Additionally, the analysis determined that this requirement was  excessively high when compared to similar types of vehicles. In fiscal year  2004, DOD’s Director of Operational Test and Evaluation (DOT&E) office  reported that overall EFV reliability remained a significant challenge  because of the system’s comparative complexity and harsh operating  environment. In 2004, The Marine Corps’ Combat Development Command  reviewed the 70-hour requirement and recommended that it be reduced to  43.5 hours. According to program officials, the primary reason for the  reduction to 43.5 hours was to more accurately depict the Marine Corps’  current mission profile for the EFV, which calls for a 12.5-hour mission  day. The Joint Requirements Oversight Council approved the reliability  reduction to 43.5 hours in January 2005.
section_title: Difficulty of Demonstrating Design Maturity Was Underestimated
paragraphs: The program’s development schedule did not allow enough time to  demonstrate maturity of the EFV design during SDD. The critical design  review was held almost immediately after SDD began. Testing of early  prototypes continued for 3 years after the decision to begin building the  SDD prototypes. Test schedules for demonstrating design maturity in the  integrated, full-system SDD prototypes proved optimistic and success- oriented, and were extended twice. After the schedules were extended,  major problems were discovered in testing the prototypes.
subsections: section_title: Best Practices for Demonstrating Design Maturity
paragraphs: Conceptually, as figure 4 illustrates, SDD has two phases: a system  integration phase to stabilize the product’s design and a system  demonstration phase to demonstrate the product can be manufactured  affordably and work reliably.
The system integration phase is used to stabilize the overall system design  by integrating components and subsystems into a product and by showing  that the design can meet product requirements. When this knowledge is  captured, knowledge point 2 has been achieved. Leading commercial  companies use several criteria to determine that this point has been  achieved, including completion of 90 percent of engineering drawings and  prototype or variant testing to demonstrate that the design meets the  requirements. When knowledge point 2 is reached, a decision review—or  critical design review—is conducted to ensure that the program is ready to  move into system demonstration. This review represents the commitment  to building full-scale SDD prototypes that are representative of the  production vehicle. The system demonstration phase is then used to  demonstrate that the product will work as required and can be  manufactured within targets. When this knowledge is captured, knowledge  point 3 has been achieved. DOD uses this conceptualization of SDD for its  acquisition policy and guidance.
The EFV program met most of the criteria for SDD critical design review,  which it held January 2001, about 1 month after entering SDD. In  particular, it had 84 percent of drawings completed and had conducted  early prototype testing during the last year of program definition and risk  reduction. However, this early prototype testing had not been fully  completed prior to critical design review. Testing of the early prototypes  continued for 3 years into SDD, well after the program office established  the SDD critical design decision to begin building the SDD prototypes.
section_title: Initial SDD Test Schedules Were Optimistic and Success-Oriented
paragraphs: The program did not allow enough time to demonstrate maturity of the  EFV design during SDD. The original SDD schedule of about 3 years  proved too short to conduct all necessary planning and to incorporate the  results of tests into design changes. Specifically, the original schedule did  not allow adequate time for testing, evaluating the results, fixing the  problems, and retesting to make certain that problems are fixed before  moving forward. Testing is the main process used to gauge the progress  provided to the customer. Consequently, it is essential to build sufficient  testing and evaluation time into program development to minimize or  avoid schedule slippages and cost increases being made when an idea or  concept is translated into an actual product. Evaluation is the process of  analyzing and learning from a test. The ultimate goal of testing and  evaluation is to make sure the product works as intended before it is  provided to the customer. Consequently, it is essential to build sufficient  testing and evaluation time into program development to minimize or  avoid schedule slippages and cost increases.
Prior to entering SDD, during both the concept evaluation and the  program definition and risk reduction phases, the EFV program conducted  a variety of component and subsystem tests. This testing included an  engineering-model and prototype-testing program, as well as modeling and  simulation test programs. Early EFV testing also included early  operational assessment tests on the initial prototype developed during  program definition and risk reduction. During this phase, the EFV program  demonstrated key aspects of performance including the technological  maturity to achieve the high water speed and land mobility needed for the  EFV mission. In addition, a number of subsystem tests were conducted on  key components of the EFV, including the main engine; water jets;  propulsion drive train components; weapons; nuclear, biological and  chemical filters; track, suspension units; and nearly all of the vehicle  electronics.
Nevertheless, the SDD schedule was extended twice to ensure adequate  system-level testing time. In November 2002, the program office extended  the test schedule by 12 months for additional testing prior to low-rate  initial production. According to program officials, this extension was  necessary for several reasons. Lessons learned from testing the early  prototypes necessitated design changes in the SDD prototypes, which  delayed delivery and testing of the SDD prototypes. In addition, testing  was taking longer than anticipated, additional time was needed for  reliability testing, and more training was required to qualify crews prior to  certain events. For example, the results of the early EFV firepower, water  operations, and amphibious ship testing revealed the need for more  testing. The schedule was delayed further to allow more time to  demonstrate the reliability of the EFV using the SDD prototypes. In March  2003, DOT&E directed that the EFV test schedule be extended for yet  another 12 months so that more developmental testing and more robust  operational testing could occur before initial production.
section_title: EFV Program Encountered Design Maturity Problems
paragraphs: After the two schedule adjustments, testing of SDD prototypes revealed  major problems in maturing the system’s design. Specifically, the program  experienced problems with the HEU, bow flap, system hydraulics, and  reliability.
subsections: section_title: Hull Electronic Unit
paragraphs: The HEU provides the computer processing for the EFV’s mobility, power,  and auxiliary computer software configuration and for the command and  control software application. Figure 5 shows the HEU.
In November 2004, during integrated system-level testing on the SDD  prototypes, there were major problems with the HEU. For example, the  water-mode steering froze, causing the vehicle to be non-responsive to the  driver’s steering inputs and both the HEU and the crew’s display panel  shut down during EFV operation. Consequently, testing ceased until the  causes of the problems could be identified and corrections made. The  program office conducted a root-cause analysis and traced the problems  to both hardware and software sources. The program office made design  changes and modifications to correct the problems, and testing resumed in  January 2005, after about a 2-month delay. According to program officials,  these changes and modifications were installed by May 2005, in the  vehicles that will be used to conduct the operational assessment tests.  Again, according to program officials, these problems have not recurred.
However, the HEU has experienced some new problems in testing since  then. For example, in June 2005, some status indicators on the crew’s  display panel shut down during land operations and had to be rebooted.  Program officials commented that corrective actions for HEU problems  have been initiated and tested to ensure that the actions resolved the  problems. We did not independently verify program officials’ statements  about initiation and testing of corrective actions.
section_title: Bow Flap
paragraphs: The bow flap is a folding appendage on the front of the EFV that is  hydraulically extended forward during EFV water operations. The bow  flap provides additional surface area that is used to generate additional  hydrodynamic lift as the vehicle moves through the water. Figure 6 shows  the bow flap.
Prior to entering SDD, major problems occurred with an earlier version of  the bow flap in testing using early prototypes. Root-cause analysis traced  these problems to bow flap overloading. Consequently, the bow flap was  redesigned but was not retested on the early prototypes before the new  design was installed on the SDD prototypes.
Problems with the new bow flaps occurred during subsequent SDD  prototype testing. For example, in September and October 2004, two bow  flaps failed—one bent and one cracked. Again, the program office  conducted a root-cause analysis, which determined that loading—while no  longer excessive—was inappropriately distributed on the bow flaps.  Following corrective action, tests were conducted in Hawaii during July to  August 2005 to validate the load capacity of the new bow flap. These tests  revealed that the design of the new bow flap needed some refinements in  order to meet the operational requirement that the EFV be capable of  operating in 3-foot significant wave heights. A program official indicated  that the test results will be used to refine the design of the new bow flap.  However, the refined bow flap design will not be tested in the  operationally required 3-foot significant wave heights until initial  operational testing and evaluation, well after the program enters low-rate  initial production.
section_title: Hydraulics
paragraphs: Hydraulic systems are key components in the EFV. For example, they  control raising and lowering the bow flap, engine cooling systems, marine  steering, and troop ramps. Hydraulic system failures are one of the top  reliability drivers in the EFV program. If the reliability requirement is to be  achieved, the myriad hydraulic problems must be resolved. The EFV has  encountered hydraulic system problems on both early and SDD  prototypes. The top four hydraulic system problems are:    Leaks from all sources, particularly leaks due to the loosening of  fittings and connectors because of vibration during EFV operations.
Various component mechanical failures experienced during EFV  testing.
Hydraulic fluid pressure spikes, particularly in the EFV’s  transmission and pumps.
Hydraulic fluid contamination by air, water, and particulates.
Program officials said that the program office has instituted a  design/test/redesign process to identify deficiencies and implement  corrections to increase vehicle reliability. According to program officials,  this process brings together the program office, contractor, various  subcontractor vendors of hydraulic components, and experts from  industry and academia to address and correct hydraulic problems as they  occur. Corrective actions thus far include:    Leaks—better sealing of connections; installation of specialized,  self-locking devices at connections most susceptible to vibration  leaks; and replacement of rigid tubing with flexible hoses to absorb  vibration.
Component mechanical failures—redesigning, strengthening, and  upgrading various parts.
Hydraulic fluid pressure spikes—reducing gear shifting during EFV  operations and installing devices to control pressure.
Hydraulic fluid contamination—flushing hydraulic systems and  instituting a variety of monitoring, maintenance, and inspection  plans to maintain hydraulic fluid and component cleanliness  requirements.
Program officials noted that corrective actions thus far have been tested to  ensure that they resolved the problems, and have been installed on the  SDD prototype vehicles. We did not independently verify this.
section_title: System Reliability
paragraphs: Based on lower demonstrated reliability and problems with early program  testing, the EFV’s reliability has not grown as planned. Expectations for  reliability are now lower, as reflected in the recent reduction to the  reliability requirement. When SDD began, the EFV was expected to  demonstrate 48 hours between failures by September 2005. Actual growth  demonstrated 28 hours between failures in August 2005. At the time of the  low–rate initial production decision now planned for December 2006,  demonstrated reliability is projected to be 38 hours between failures. The  original and current reliability growth curves for the EFV are shown in  figures 7 and 8, respectively.
In comparing the planned and actual reliability growth curves, it is clear  that the actual test hours accumulated have been significantly less than  planned. In fact, the original plan called for conducting 12,000 hours of  testing by the original September 2005 production decision; according to  the current plan, test hours will not reach this level until early 2008. The  reduction in test hours is due, in part, to the other problems that occurred  in testing. The accumulation of test hours is significant for reliability. In  general, reliability growth is the result of an iterative design, build, test,  analyze, and fix process. Initial prototypes for a complex product with  major technological advances have inherent deficiencies. As the  prototypes are tested, failures occur and, in fact, are desired so that the  product’s design can be made more reliable. Reliability improves over time  with design changes or manufacturing process improvements.
The program office acknowledges that even with the changes in mission  profile and reduction in the operational requirement, reliability for the  EFV remains challenging. In addition, the most recent DOT&E annual  report found that the EFV system’s reliability is the area of highest risk in  the program. DOT&E has reviewed the EFV’s current reliability growth  plan and believes that it is realistic but can only be validated during initial  operational testing and evaluation in 2010.
According to the program manager, an additional 15 months would have  been needed for more robust reliability testing, production qualification  testing, and training, after the program entered low-rate initial production  in September 2005, as originally planned. The March 25, 2005, rebaselining  extended the schedule by 24 months and postponed low-rate initial  production until September 2006, which has now been extended to  December 2006. While DOD’s December 2004, Program Budget Decision  753 served as the catalyst for this rebaselining, the program manager  stated that he probably would have asked for a schedule extension of   15 months after entering low-rate initial production in September 2005,  even if the budget decision had not occurred. DOD and Marine Corps  officials verified that, although the program manager did not officially  request this 15-month extension, he had been discussing an extension with  them before the budget decision was issued. However, to the extent that  the extra 9 months resulting from the budget decision prove unneeded for  program management reasons, they will be an added cause for schedule  and cost growth.
section_title: Risks Remain for Demonstrating Design and Production Maturity
paragraphs: Three areas of risk remain for demonstrating design and production  maturity, which have potential cost and schedule consequences—risks to  the EFV business case. First, while the EFV program has taken steps and  made plans to reduce risk in the production phase, production risk  remains in the program. Current plans are to enter low-rate initial  production without requiring the contractor to ensure that all key EFV  manufacturing processes are under control. Second, the EFV program will  transition to initial production without the knowledge that software  capabilities are mature. Third, two key performance parameters— reliability and interoperability—are not scheduled to be demonstrated  until the initial operational test and evaluation phase in fiscal year 2010,  about 4 years after low-rate initial production has begun. The program  office has developed plans to resolve performance challenges and believes  it will succeed. However, until the plans are actually implemented  successfully, the EFV’s design and production maturity will not be  demonstrated and the potential for additional cost and schedule increases  remains.
subsections: section_title: Manufacturing Process Maturity Problems
paragraphs: While the EFV program has taken steps and made plans to reduce risk in  the production phase, production maturity risk remains in the program.  Current EFV program plans are to enter low-rate initial production  without requiring the contractor to ensure that all key EFV manufacturing  processes are under control, i.e., repeatable, sustainable, and capable of  consistently producing parts within the product’s tolerance and standards.  Establishing such control is critical to ensuring that the EFV can be  produced reliably and without unexpected production problems. In  addition, DOD’s system acquisition policy provides that there be no  significant manufacturing risks prior to entering low-rate initial production  and that manufacturing processes be under statistical process control  prior to starting full-rate production.
Leading commercial firms rely on statistical process control to ensure that  all key manufacturing processes are under control before they enter  production. Statistical process control is a technique that focuses on  reducing variations in manufactured parts, which in turn reduces the risk  of entering production with unknown production capability problems.  Reducing and controlling variability lowers the incidence of defective  parts and thereby products, which may have degraded performance and  lower reliability. Defects can also delay delivery and increase support and  production costs by requiring reworking or scrapping. Consequently, prior  to entering production, leading commercial firms collect and analyze  statistical process control data. Leading commercial firms also use a  measure of process control called the process capability index to measure  both the consistency and the quality of output of a process. DOD’s  acquisition policy applies a lower standard. It provides that there be no  significant manufacturing risks prior to entering low-rate initial production  and that manufacturing processes be under statistical process control  prior to starting full-rate production.
The EFV program is working toward the DOD standard. EFV program  officials said that statistical process control will not be used to ensure that  all key EFV manufacturing processes are under control prior to entering  low-rate initial production. They stated that they have taken actions to  enhance EFV production readiness. For example, they noted that one of  the most important risk mitigating actions taken was ensuring that SDD  prototypes were built using production-representative tooling and  processes. Program officials also believe that production process maturity  will be demonstrated by achieving repetitive schedule and quality  performance during low-rate initial production. In addition, the program  plans to collect statistical process control data during low-rate initial  production to track equipment and machine performance and detect  statistical shifts. The program believes that using statistical process  control data in this manner will result in earlier detection of machine  malfunctions. Program officials told us that once sufficient quantities of  the EFV are produced and baseline statistical process control data  collected, the results of the analyses of this data will be implemented for  any production measurements that demonstrate process stability. The  program office believes that this approach will allow for use of statistical  process control for implementation of stable manufacturing processes  during low-rate initial production. However, the program office does not  plan to set and achieve a process capability index for the EFV production  efforts.
The actions taken by the program may help to mitigate some production  risk. In fact, EFV’s plan to collect and use statistical process control data  goes further than what we have found on most DOD weapon system  programs. However, these actions do not provide the same level of  confidence as having the manufacturing processes under statistical  process control before production. The EFV program’s approach of  foregoing such control increases the risk of unexpected production  problems during manufacturing. This risk is compounded by the fact that  plans call for reliability and interoperability, along with resolution of other  technical problems, to be operationally tested and demonstrated during  low-rate initial production, not before.
section_title: Software Development Capability Maturity Problems
paragraphs: Under current plans, the EFV program is at risk of entering low-rate initial  production before software development capabilities are mature. Again,  leading commercial firms ensure that software development capabilities  are mature before entering production in order to prevent or minimize  additional cost growth and schedule delays during this phase.
Furthermore, DOD’s weapon system acquisition policy calls for weapon  systems to have mature software development capabilities before they  enter low-rate initial production.
In assessing software capability maturity, commercial firms, DOD, and  GAO consider the software capability maturity model developed by  Carnegie Mellon University’s Software Engineering Institute to be an  industry standard. This model focuses on improving, standardizing, and  certifying software development processes, including key process areas  that must be established in the software developer’s organization. The  model is essentially an evolutionary path organized into five maturity  levels:    Level 1, Initial—the software process is ad hoc and occasionally  chaotic. Few processes are defined, and success depends on  individual effort.
Level 2, Repeatable---basic project management processes are  established to track cost, schedule, and functionality. The necessary  process discipline is in place to repeat earlier successes on projects  with similar applications.
Level 3, Defined—the software process for both management and  engineering activities is documented, standardized, and integrated  into a standard process for the organization. All projects use an  approved, tailored version of the organization’s standard process  for developing and maintaining software.
Level 4, Managed—Detailed measures of the software process and  product quality are collected. Both the software development  process and products are quantitatively understood and controlled.
Level 5, Optimizing—Continuous process improvement is enabled  by quantitative feedback from the process and from plotting  innovative ideas and technologies.
The EFV program has had problems with maturing its software  development capabilities. The EFV’s prime contractor, General Dynamics  Land Systems (GDLS), which at the time had a level 3 maturity software  capability, developed all software for the early EFV program. According  to the program office, when the program entered SDD, responsibility for  EFV’s software development was transferred to GDLS’ amphibious  development division, General Dynamics Amphibious Systems (GDAMS).  GDAMS has a level 1 maturity software capability. Consequently, the SDD  contract required GDLS to achieve a software development capability  maturity level 3 for all EFV software contractors and subcontractors  within 1 year of the contract award date, July 2001. In January 2002, the  program extended this requirement by 1 year, until July 2003.  Nevertheless, while GDAMS twice attempted to achieve level 3 software  development capability maturity, it did not succeed.
Program officials considered GDAMS’s inability to achieve an acceptable  level of software development capability maturity a risk to the program.  To mitigate this risk, in January 2004, the program manager began  developing a risk mitigation plan. As part of this plan, representatives from  the EFV program office, GDAMS, and Ogden Air Logistics Center’s 309th  Software Maintenance Group—a certified level 5 maturity software  development organization—formed a Software Partnership Working  Group to address software development capability maturity issues. As of  February 2006, EFV program officials were in the process of negotiating a  memorandum of agreement with the 309th Software Partnership Working  Group to develop the EFV’s low-rate initial production software. The   309th will work in partnership with GDAMS as specified by the terms of the  memorandum of agreement. Its involvement is to ensure that the EFV’s  software development capability will be at the desired maturity level.
However, the 309th Software Maintenance Group will not complete the  software development for the EFV’s low-rate initial production version  until September 2006. Furthermore, GDAMS does not plan to insert this  software into the EFV vehicles until fiscal year 2008, well after low-rate  initial production has begun. This means that the low-rate initial  production decision will be made without the integration of mature  software. Furthermore, the software itself will not be demonstrated in the  vehicle until well into low-rate initial production. While the program office  believes that the level of software risk is an acceptable level risk, we have  found that technology—including software—is mature when it has been  demonstrated in its intended environment.
While involving the 309th Software Maintenance Group helps to mitigate  the risk of immature software development capability in the EFV program,  it increases certain other risks. The memorandum of agreement distributes  the responsibility for software development between the three  participants. However, much of the responsibility for developing a  working software package in an acceptably mature environment shifts  from the prime contractor to the Marine Corps. The software will now  become government-furnished equipment or information. In essence, the  Marine Corps has now assumed much of the risk in the software  development effort. If the software does not work according to the  requirements, it will be incumbent upon the Marine Corps—not the prime  contractor, GDLS—to correct the problems. Furthermore, if the  integration of the government-furnished software into the vehicles creates  additional problems, the Marine Corps could be responsible for  corrections. Both of these situations could lead to cost and schedule  growth, and thus increase risks to the program.
section_title: Performance Challenges Not Yet Fully Resolved
paragraphs: Several EFV performance challenges are not yet fully resolved.  Specifically, a key performance parameter—interoperability—cannot be  properly demonstrated until initial operational testing and evaluation in  fiscal year 2010, well after low-rate initial production has begun.  Interoperability means that the EFV communication system must provide  essential command, control, communications, and intelligence functions  for embarked personnel and EFV units. In addition, the EFV  communication system must be compatible—able to communicate—with  other Marine Corps systems as well as with Army, Navy, Air Force, and  North Atlantic Treaty Organization systems. In order to demonstrate  interoperability, the EFV must participate in operational tests that involve  these joint forces. Another key performance parameter—reliability—has  been problematic and still presents a significant challenge. It also is not  scheduled to be demonstrated until initial operational testing and  evaluation. Furthermore, the bow flap has been problematic and, while  improved, still requires some design refinement and has not yet been  successfully tested at its operational performance level.
Program officials commented that they have developed plans to resolve  remaining EFV performance challenges and are optimistic that these plans  will be implemented effectively and testing successfully completed.  However, there are no guarantees that this will actually happen.  Consequently, the performance challenges remain risks to the program  until they are fully resolved with effective solutions actually demonstrated.
section_title: Conclusions
paragraphs: The EFV has encountered risks to its business case because of problems  encountered in full-system testing, coupled with an SDD schedule that did  not allow enough time for conducting the testing and learning from it.  Using the lens of a knowledge-based business case, the start of SDD was  sound on requirements and technology maturity (knowledge point 1).  While design stability was judged to be attained at the critical design  review (knowledge point 2) immediately after entering SDD, it appears  that holding critical design review so soon was premature. The acquisition  strategy did not provide the resources (time and money) necessary to  demonstrate design maturity and production maturity (knowledge point  3). However, we do note that the EFV program is planning to do more with  statistical process control than most other programs we have reviewed.
In retrospect, the EFV program would have been more executable had the  SDD phase allowed for completion of early prototype testing before  holding the SDD critical design review and committing to building the SDD  prototypes. Another lesson learned is that while it is necessary to  demonstrate one knowledge point before a subsequent one can be  demonstrated, this alone is not sufficient. Attaining one knowledge point  does not guarantee the attainment of the next one. Rather, the acquisition  strategy for any program must adequately provide for the attainment of  each knowledge point even in programs, such as the EFV, which were in a  favorable position at the start of SDD.
The EFV program has put into place a number of corrective actions and  plans to overcome and mitigate weaknesses in acquisition strategy.  Nevertheless, design, production, and software development capability  maturity have not yet been fully demonstrated and technical problems  fully corrected. It is important for the business case for the EFV to remain  valid in light of these changes and that the remainder of SDD adequately  provide for the demonstration of design, production, and software  development capability maturity before committing to production.
While these problems must be acknowledged and addressed, the fact that  the EFV program has had a number of sound features should not be  overlooked. In this vein, the program can still be the source of lessons that  DOD can apply to other programs. In particular, it is important that all of  the elements of a sound business case be present at the start of SDD.  While it is generally recognized that missing an early knowledge point will  jeopardize the remaining ones, it must also be recognized that later  knowledge points are not guaranteed even if early ones are achieved. If the  acquisition strategy does not adequately provide for the attainment of all  knowledge points, the estimates for cost and schedule will not have a  sound basis.
section_title: Recommendations for Executive Actions
paragraphs: We are recommending that the Secretary of Defense ensure that:    EFV design, production, and mature software development  capabilities are demonstrated before Milestone C;   adequate resources are available to cover such demonstration and  provide for risks; and the business case for EFV (including cost and expected capability),  after including the above, still warrants continued investment.
We also recommend that the Secretary of Defense draw lessons learned  from EFV and apply them to the Defense Acquisition University’s  curriculum for instructing program executives, managers, and their staffs.  Such lessons might include understanding that attaining one knowledge  point does not guarantee the attainment of the next one; the importance of  having a sound business case for each phase of development; the right  time to hold a critical design review; and the importance of allowing  sufficient time to learn from testing.
section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on a draft of our report, DOD’s Acting Director for Defense  Systems concurred with our recommendations. In doing so, DOD stated  that the Department currently plans to assess the readiness of the EFV  program for a low-rate initial production decision within a year. This  assessment will review the maturity of the EFV design, including software,  its production readiness for low-rate initial production, and its  demonstrated capability, as well as program costs and risks. Continued  investment in EFV will be based on that information. The full text of the  department’s response is in appendix II.
The Department notes that our best practices construct for production  readiness is difficult to reconcile with its current acquisition production  decision points. World class companies we have visited do, in fact, often  have a limited production run that they use to manufacture a small  number of production representative assets; however, they do not make a  decision to invest in the tooling necessary to ramp up to full production  until after those assets have been tested by the customer and their critical  manufacturing processes are in control. DOD’s low-rate initial production  decision reflects the decision to invest in all of the resources needed to  achieve full-rate production. We believe this is too soon and that DOD  would benefit from this lesson by focusing low-rate initial production on  demonstrating the product and process and waiting to invest in more  resources, such as tooling, to ramp up until the full-rate production  decision has been made.
We are sending copies of this report to the Secretary of Defense, Secretary  of the Navy, and other interested parties. We will also provide copies to  others on request. In addition, the report will be available at no charge on  the GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  on (202) 512-4841. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  GAO staff who made major contributions to this report are listed in  appendix III.
Paul L. Francis  Director, Acquisition and Sourcing Management.
section_title: Appendix I: Scope and Methodology
paragraphs: To assess the current status of the EFV (particularly the status of the  production decision), the factors that contributed to the current status,  and future risks in the program, we interviewed key officials from DOD’s  Director, Operational Test and Evaluation, the Office of the Secretary of  Defense’s Program Analysis and Evaluation office, the U.S. Marine Corps,  Isothermal Systems Research, Inc., in Washington, D.C., and the 309th  Software Maintenance Group, in Ogden, Utah. We also interviewed the  Direct Reporting Program Manager for the EFV and the prime contractor,  General Dynamics Land Systems, in Woodbridge Virginia. We examined  and analyzed pertinent program documentation, including the Selected  Acquisition Reports; Test and Evaluation Master Plan; Developmental  Testing Schedule; Budget Justification documents, Program Management  Plan; Acquisition Strategy Plan; DOD’s Operational Testing, and  Evaluation reports; Operational Requirement Documents, and the  Software Development Plan. We relied on previous GAO work as a  framework for knowledge-based acquisition.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgement
subsections: section_title: GAO Contact
section_title: Acknowledgements
paragraphs: In addition to the contact named above, D. Catherine Baltzell, Assistant  Director; Leon S. Gill; Danny Owens; Steven Stern; Martin G. Campbell;  and John Krump made key contributions to this report.