Defense Acquisitions: Steps to Improve the Crusader Program's Investment Decisions
Released: 2002-02-25T00:00:00

Letter

Background

The Army plans to invest about $11 billion developing and procuring the Crusader, an automated, next generation field artillery system. To date, the program has spent about $1.7 billion in development costs. It plans to procure 482 Crusader systems—each system consisting of a self-propelled 155-millimeter howitzer and a resupply vehicle. The Army is developing 2 different resupply vehicles—1 with tracks and 1 with wheels—and plans to procure 241 of each type. The purpose of the Crusader system is to overcome threats from enemy artillery and reconnaissance or surveillance systems as well as have the mobility needed to keep up with Army tanks and fighting vehicles. Figure 1 shows the planned Crusader howitzer, figure 2 the planned tracked resupply vehicle, and figure 3 the planned wheeled resupply vehicle.

The Army restructured the Crusader program in January 2000 to align Crusader’s design with the Army’s transformation to a lighter force. The Army’s transformation will affect all aspects of Army organization, training, doctrine, leadership, and strategic plans as well as the types of equipment and technology the Army acquires. The Army expects the transformation to be at least a 30-year process and has not estimated its full cost. The centerpiece of the lighter, more deployable future force is the Future Combat Systems. The Future Combat Systems concept is a system of ground and air, manned and unmanned weapon systems, each under 20 tons that is planned to replace most, if not all, of the Army’s ground combat systems without a loss in lethality and survivability. Artillery systems are among those to be replaced.

The Army expects the Crusader system to fill the existing gap in artillery capabilities until it is replaced by the Future Combat Systems. In keeping with the transformation philosophy of lightweight vehicles and ease of deployability, the Army is redesigning Crusader to make it lighter and more deployable, with the goal of reducing the weight of the self-propelled howitzer and tracked resupply vehicle from about 60 tons to about 40 tons each. Program officials said that a lighter system would enhance operational flexibility in employing Crusader in support of any operation.

The Crusader is currently in the program definition and risk reduction phase of its development program. In April 2003, the program is scheduled for a milestone B review to determine whether it is ready to enter its system development and demonstration phase. Milestone B is the point at which DOD decides whether to commit major resources to develop and design the system and to demonstrate its integration, interoperability, and utility. The milestone marks the start of the program’s product development. The Army plans to deliver the first full Crusader prototype system in October 2004, followed by a low-rate initial production decision in February 2006, and initial system fielding in April 2008.

Critical Technologies Need Additional Maturity to Better Assure Low-risk Product Development

Based on current Army plans, the Army will begin the Crusader’s product development in April 2003 but before maturing critical Crusader technologies to a level considered low risk relative to best practices. These risks relate less to whether these technologies can be matured, but more to how much time and cost it will take to mature them. If, after starting product development, the Crusader technologies do not mature on schedule and instead cause delays, the Army may spend more and take longer to develop, produce, and field the Crusader system. Crusader performance goals may also be at risk. On the other hand, the Army has made improvements to the management of the Crusader software development process.

Assessing Technology Readiness Provides Opportunities to Improve Outcomes

The maturity of a program’s technologies at the start of product development is a good predictor of that program’s future performance. Our past reviews of programs incorporating technologies into new products and weapon systems showed that they were more likely to meet product objectives when the technologies were matured before product development started. For example, the Ford Motor Company’s practice of demonstrating new technologies in driving conditions before they are included in a new product is essential to ensuring that the new product can be developed on time and within budget. Similarly, we have found that the early demonstration of propulsion and water-planing technologies, essential to the performance of the Marine Corps’ Advance Amphibious Assault Vehicle, has been instrumental to that program’s staying within 15 percent of cost and schedule estimates.

Conversely, cost, schedule, and performance problems were more likely to occur when programs started with technologies at lower readiness levels.For example, the enabling technologies for the Army’s Brilliant Anti-Armor Submunition program were very immature at the start of the program, and their delays became major contributors to the program’s subsequent 88-percent cost growth and 62-percent schedule slippage. Separating technology development from product development into two distinct program phases is a best practice of both successful commercial and defense programs. This entails demonstrating all critical technologies at the component or subsystem level in an operational environment during technology development, prior to committing major funding to product development. Under this practice, the critical technologies would be demonstrated in component or subsystem prototypes that are nearly the right size, weight, and configuration needed for the intended product. Such demonstrations need not require a full system prototype of a Crusader vehicle, but can be done using surrogate vehicles.

Technology readiness levels (TRL) are a good way to gauge the maturity of technologies. TRLs were pioneered by the National Aeronautics and Space Administration to determine the readiness of technologies to be incorporated into products such as weapon systems. Readiness levels are measured along a scale of one to nine, starting with paper studies of the basic concept, proceeding with laboratory demonstrations, and ending with a technology that has proven itself on the intended product. TRLs are based on actual demonstrations of how well specific technologies perform in the intended application. For example, a technology that has been demonstrated in an operational environment using subsystem prototype hardware (such as a complete cannon system) that is at or near the final system design would be rated as a TRL 7. The individual TRL descriptions can be found in appendix I.

DOD has agreed that technology readiness assessments are important and necessary in assisting officials who decide when and where to insert new technologies into weapon system programs. In January 2001, DOD issued a new acquisition instruction that redefined the phases in the defense acquisition cycle and emphasized the role of technology development in the acquisition process. Under the instruction, programs use the concept and technology development phase, which precedes the system development and demonstration phase, for developing components and subsystems that must be demonstrated before integration into the system. The first portion of system development and demonstration phase is dedicated to integrating the components and subsystems into the system. The instruction states that DOD prefers that technology be demonstrated in an operational environment but must be demonstrated in a relevant environment to be considered mature enough for product development in the system development and demonstration phase. According to the TRL descriptions, technology demonstrated in an operational environment is TRL 7 and technology demonstrated in a relevant environment is TRL 6.

Maturing technology from a TRL 6 to a TRL 7 represents a major step up in maturity. A technology at the TRL 6 maturity level needs only to be demonstrated as a subsystem prototype or model in a laboratory or simulated operational environment. A technology at the TRL 7 maturity level must be demonstrated as a subsystem prototype at or near the size of the required subsystem outside the laboratory in an actual operational environment. For example, operating a prototype engine on a laboratory test stand that simulates the effects of the vehicle’s weight on the engine would be a TRL 6 level demonstration while operating an engine in a surrogate vehicle or actual prototype that weighed 50 tons, on roads and cross country, would be a TRL 7 demonstration.

In June 2001, DOD issued a new acquisition regulation. It stated that technology maturity is a principal element of program risk and directed technology readiness assessments for critical technologies sufficiently prior to selected milestone decision points—including milestone B--to provide useful technology maturity information to the acquisition review process. Although the new regulation recognizes that TRLs enable consistent, uniform discussions of technical maturity across different types of technologies and provides the definitions of TRLs used in this report, it permits the use of TRLs or “some equivalent assessment” when performing a technology readiness assessment.

Technology Maturation Will Continue into Product Development

In June 2001, Crusader program office engineers and we assessed the maturity of 16 critical Crusader technologies using TRLs. This joint assessment determined that 10 of the 16 critical Crusader technologies were below TRL 7. Since the Crusader program is not scheduled to commit to product development until April 2003, the Army still has time to mature the 10 critical technologies to a TRL 7 level—demonstrate them in a component or subsystem prototype in an operational environment. However, the Army’s Crusader plans will result in 10 of the critical Crusader technologies remaining below TRL 7 at the milestone B decision and in technology development continuing into the product development phase. As a result, the Crusader program would not reach the low levels of risk that best practices show is needed for meeting product development cost and schedule commitments. Table 1 shows the results of our joint technology readiness assessment.

As shown in table 1, if technology develops as planned, eight critical technologies will be at a TRL 6 level of maturity and two will be at a TRL 5 level of maturity at milestone B. While some technologies may embody some risk in meeting requirements, for the most part, the risk in the Crusader technologies involves the amount of time and effort needed to reach maturity. The planned technology maturity levels for the Crusader program at milestone B increase the probability that technical problems, if they occur, will need to be resolved in the higher cost environment of system development and demonstration. Confining delays in maturing technology to a time prior to the start of product development—in an environment where small teams of technologists work in laboratories and are dedicated to perfecting the technology—is critical to saving time and money. Conversely, if delays occur in product development when a large engineering force is in place to design and manufacture the product, delays would be much more costly. In fact, industry experts estimate that a delay during product development costs several times more than a similar delay that occurs before product development.

Under the current Crusader acquisition plans, the critical technologies would be demonstrated in two steps after milestone B. Program officials are planning to demonstrate mobility component technologies first and then the remaining critical technologies. They recognize a risk in integrating the Crusader’s mobility components—track, suspension, engine, and transmission—and plan to produce a mobility test rig to demonstrate that integration and to start accumulating reliability data on the mobility components. The mobility test rig would have the additional advantage of demonstrating the maturity of those technologies in an operational environment. The contractor is scheduled to deliver the mobility test rig in December 2003. The test rig would later be rebuilt as a Crusader prototype.

The remaining critical technologies would not be demonstrated until after the contractor delivers the Crusader prototypes. The first Crusader system prototype is scheduled for delivery in October 2004 and is to enter testing the same month. Other prototypes would enter testing as they are delivered. The Army plans to award contracts for low-rate initial production long-lead items in March 2005—less than a fourth of the way through the prototype-testing schedule. This leaves little time in the Crusader’s projected system development and demonstration schedule for solving unanticipated problems before the Army awards contracts for long- lead production items.

The Army’s approach to readying the Crusader for milestone B is to demonstrate progress toward achieving five of the system’s requirements, two of which are key performance parameters—the cannon rate of fire and the ability to resupply the self-propelled howitzer. For example, a Crusader key performance parameter is that the Crusader cannon be able to fire 10 to 12 rounds per minute; however, the program only needs to demonstrate the ability to fire 6 rounds per minute before milestone B. The demonstrations, called exit criteria, were approved by both the Army and DOD. Among the demonstrations required by the exit criteria, only the cannon system is expected to be demonstrated in an operational environment; the other critical technologies are expected to be demonstrated in a laboratory environment.

Moreover, like many other DOD programs, the Crusader program is using risk management plans and engineering judgment, without the benefit of TRLs, to assess technological maturity and mitigate program risk. Risk management plans and engineering judgment are necessary to manage risk in any major development effort like the Crusader. However, we have found in our reviews that without an underpinning, such as TRLs, that allows transparency into program decisions, significant technical unknowns may be judged acceptable risks because a plan exists for resolving them. For example, we recently reported that while DOD judged the technical risks facing the Joint Strike Fighter as acceptable for starting product development, an analysis of TRLs showed that eight critical technologies were below TRL 7, with six technologies at TRL 4 or 5. When problems are encountered in resolving these unknowns, programs often fail to meet promised outcomes, as noted above with the Brilliant Anti-Armor Submunition program.

Army Has Improved the Crusader’s Software Development Management

The Army has made improvements to its management of the software development process. Program officials stated that they would continue to aggressively manage the software development program to achieve and sustain the software process improvements.

The automated Crusader system will be a software intensive program, projected to use about 1.9 million lines of code. Unlike any previous ground vehicle, all of the major functions of the Crusader are automated, including aiming, loading, and firing the cannon; managing inventory (projectiles and propellant); and resupplying the howitzer with ammunition and fuel. The crew compartment consists of a digital command center, with flat panel displays and re-configurable crew stations that give the crew real-time situation awareness, targeting information, integrated electronic technical manuals, decision aids, and diagnostic information.

In 1998, the program began to experience software problems before meeting the software’s preliminary design milestone. In June 1999, the Army decided that there were incomplete areas of the preliminary design and that the software team was not resolving design issues in a timely manner. Additionally, the software engineering team lacked disciplined quality assurance and configuration management practices, which led to some of the problems.

In response, the program office tasked a software action team to identify problems and recommend improvements. The team drafted a recovery plan and recommended a number of process improvements for the prime contractor to implement. Program officials used the Software Development Capability Maturity ModelSM to define and determine the software development process maturity. The Software Engineering Institute, part of Carnegie Mellon University, developed the model to measure and rank an organization’s software development and acquisition process. The contractor agreed to mature its software engineering processes to a level where the standard processes for software development, such as project and risk management, are documented and enforced across the organization. According to the Software Engineering Institute, increasing the maturity level of an organization’s software engineering process puts the organization in better position to successfully develop software.

As a result of these efforts, the Army and its prime contractor have made improvements to their management of the Crusader software engineering process. Improved areas include requirements generation and validation, quality assurance, configuration management, risk management, schedule and cost estimation, project tracking and control, and peer reviews of software engineering products such as design documents, code, and test plans. In addition, outside experts assisted in software analysis and design. Others were brought in to independently assess the software recovery plan. The contractor implemented a number of changes in the software design process, including the establishment of a common set of software development and management tools shared by all software teams and improved software testing. The program office has also revised the Crusader contract to provide the contractor monetary incentives to produce high-quality software on schedule. Software teams are also tracking progress and reporting it to management on a weekly or biweekly basis and have greatly improved their processes for estimating the size and schedule of the software. As a result of these improvements, the contractor has made more timely deliveries of software.

Army officials will need to continue their aggressive management approach because significant amounts of software remain to be developed before the Crusader is fully operational. Program officials stated that they would continue to manage the program to achieve and sustain the software process improvements.

Lighter-weight Crusader May Not Significantly Improve Strategic Deployability

The Army has made considerable progress over the past 2 years in redesigning the Crusader to substantially reduce its size and weight. In general, a lighter system offers a number of advantages, such as lower fuel consumption and easier transportation by truck and rail. However, it is uncertain that the requirement to deploy two Crusader howitzers on a C-17 aircraft provides a significant improvement in strategic deployability. Efforts to meet the deployability requirement will be a challenge and may require costly design changes and/or performance tradeoffs.

According to an Army official, in October 1999, the Chief of Staff of the Army directed that the Crusader system become lighter and more deployable to better fit in with the Army’s transformation to lighter forces. The Army subsequently revised the Crusader's Operational Requirements Documents to reflect new deployability requirements. Specifically, the documents state that the Crusader vehicles must not exceed 42 tons at curb weight and 50 tons at combat weight; any combination of two Crusader vehicles, at curb weight, must be air transportable on both a C-5 and a C-17 aircraft; and both the C-5 and C-17 aircraft must be able to transport a single Crusader vehicle at combat weight.

Crusader’s Reduced Size and Weight May Not Provide a Significant Improvement in Deployability

The main reason for the decision in January 2000 to restructure the program and redesign the Crusader weapon system was to reduce the system’s weight and to improve its strategic deployability by air. However, the Army expects to rarely airlift the Crusader system—only during extreme emergencies—and that, in those circumstances, it would be likely that only small numbers of Crusader systems would be airlifted. Sealift would be the primary means of moving the Crusader system over long distances. In February 1999, the Army reported to Congress that the fielding of a lighter-weight Crusader would provide little in improved strategic deployability over a heavier version. In May 2000, the DOD’s Office of Program Analysis and Evaluation questioned the need to improve the Crusader’s deployability, stating that it is unclear whether airlifting a small force of the heavier Crusaders, when needed, would be a severe burden on airlift.

A limited Army analysis comparing the deployability by air of small numbers of the original heavier Crusader with that of the lighter-weight Crusader showed that the lighter-weight Crusader system might not significantly improve the system’s strategic deployability. For example, this analysis showed that the lighter-weight Crusader system would reduce the number of sorties required to carry two Crusader systems and support equipment by 20 percent—one aircraft sortie—over the system’s original, heavier design. The study showed that it would take four C-17 sorties to airlift two of the lighter-weight Crusader systems and support equipment while it would take five sorties to airlift two of the original heavier systems and support equipment. In addition, the heavier Crusader howitzers and both resupply vehicles would arrive loaded for combat while the lighter Crusader howitzers and only one resupply vehicle would arrive loaded for combat. The other resupply vehicle would have to be manually loaded upon arrival.

The recent analysis was done with inputs from various Army officials but has not been officially reviewed by the Air Force. Prior to our request, the Army had not formally analyzed the improvements in strategic deployability offered by a 40-ton Crusader over the earlier 60-ton Crusader.

Army Faces Risks in Meeting Crusader’s Deployability Requirement

Meeting the requirement for carrying two Crusader howitzers on a C-17 aircraft will be challenging. According to the Air Force, the C-17 aircraft is a more versatile aircraft and smaller than the C-5 aircraft. The C-5 is normally used for strategic deployments—into and out of the combat theater—while the C-17 aircraft can be used for both strategic deployments and tactical missions within a combat theater. According to Army and Air Force officials responsible for aircraft loading plans, the only possible way to load two Crusader howitzers on a C-17 aircraft would be back to back. However, they have concerns about this loading method. First, it will be a very tight fit with one howitzer’s cannon barrel expected to be 20 inches from the forward bulkhead (on the edge of a crew safety zone) and the other howitzer’s barrel expected to be within 3 inches of the stowed aft loading ramp. Second, according to an Air Force official, the 59 inches separating the two howitzers may not be enough room to properly restrain the vehicles with heavy chains.

In October 2001, the Army performed a preliminary computer analysis of loading two Crusader howitzers on a C-17. It indicated that, if the vehicles dimensions remain the same through redesign, development, testing, production, and fielding, the two howitzers may fit. This analysis also showed that the loading plan would be a very tight fit and does not address the issue of restraining the howitzers during flight. Air Force officials have not reviewed this analysis. Army and Air Force officials told us that it is unlikely they will know if the Crusader can actually be loaded and carried until two lighter-weight prototypes are produced and tested in a C-17 aircraft.

Army officials told us that, if carrying two Crusaders on a C-17 aircraft is not feasible, they will still accept the Crusader system because it is a much more capable system than the current self-propelled howitzer system, the Paladin. Program officials also told us that reducing the system’s weight is desirable because it reduced the logistics needed to support the system and improves, among other things, ground transportability and mobility.

Cost and/or Performance Trade-offs May Be Needed to Meet Weight Requirement

According to the DOD and the Army, achieving the Crusader’s reduced weight requirement and meeting the 42-ton limit will be a difficult challenge and will require aggressive weight management to mitigate the risks involved with system weight. As of November 2001, the Crusader howitzer is projected to weigh 41.2 tons, which is close to the upper limit of the 42-ton curb weight requirement. This projection, however, is based on computer modeling that is still evolving. The projected weight could change considerably as specific components are fabricated and tested. Program office officials told us that, at this point in time, they have an 80-percent confidence level in the model’s weight projection.

The Army has already made significant changes to the Crusader system design to reduce the curb weights of the system’s vehicles. The curb weight of the howitzer is expected to go from 60 tons to a projected weight of below 42 tons. To achieve this weight reduction, the program office is redesigning the Crusader system by reducing the size and payload of the Crusader vehicles, substituting lighter weight materials for some components, and developing, with the Abrams tank program, a lighter weight engine. Additionally, the team plans to remove the heavy armor for top attack and road wheel protection and make it into kits that can be applied when needed in combat situations. To help reduce the overall weight of the Crusader system, the team decided to use a Palletized Load System truck carrying a newly designed resupply module as a second type of Crusader resupply vehicle—a wheeled resupply vehicle.

Although the Army has not made vehicle weight a key performance parameter for the Crusader program, it has instituted an aggressive weight management program designed to mitigate the risks associated with maintaining the 42-ton per vehicle weight limit. As part of the weight management program, the Army may have to consider the trade-offs between the system’s weight and the program’s cost, schedule, and performance requirements in order to achieve the required curb and combat weights. The program is also in the position of not being allowed any weight growth during development, production, fielding, and service. Before the Crusader redesign, the program had a 17-percent weight growth expectation for the Crusader vehicles. According to an Army official, if a new capability is added to the Crusader that increases its weight, the Army will have to find a way to reduce the weight of the Crusader by an equivalent amount.

Apparent Overlap of Crusader and Future Combat Systems Programs Creates Uncertainties

The Army’s current schedule to begin fielding the Crusader system and its replacement, the Future Combat Systems, in the same fiscal year—2008— represents a potential risk of investing in duplicative systems to fulfill the same missions. However, at this time it is uncertain that the initial versions of the Future Combat Systems will have the capabilities to meet the Crusader’s missions.

The Future Combat Systems are expected to be revolutionary, lightweight weapon systems—20 tons or less—that involve manned and unmanned, ground and air systems, all of which would be digitally networked together. All the vehicles in the system are being designed for transport on a C-130 or similar aircraft—which are smaller aircraft than the C-17. Future Combat Systems vehicles may include command and control systems, reconnaissance systems, direct- and indirect-fire guns, rockets, and antitank missiles.

The Future Combat Systems program is in an earlier stage of development than the Crusader—it is still in its initial 2-year concept design. Although the Future Combat Systems is a complex system of systems and the Army is still developing system concepts and technologies, the Army expects that the Future Combat Systems can be developed and produced in much shorter time frames than other weapons programs. Under the current Army schedule, the initial versions of the Future Combat Systems might enter the system development and demonstration phase as early as fiscal year 2003 and the first combat unit is scheduled to be equipped in 2008. Once fully fielded, the Future Combat Systems are intended to replace all of the Army’s heavy weapon systems including the Crusader. Current Army plans show the Crusader to be in the force until 2032 or later.

Because all the technologies needed for the Future Combat Systems may not be mature enough to be put into systems, the Army is planning to develop the initial version of the Future Combat Systems with less than its full capabilities and then upgrade it in a number of steps, called blocks, as the required technologies mature. The Army has not defined the capabilities that it can develop in the initial version of the Future Combat Systems, which it hopes will enter product development in 2003. As early as February 2002, the Army plans to award a contract to define these initial capabilities based on technologies that are mature enough to enter system development and demonstration in 2003.

Eventually, the Army expects the Future Combat Systems to meet, using advanced technologies, the same artillery missions as the Crusader and eventually replace the Crusader system. While the final weapon technologies have not been selected for the Future Combat Systems, technologies that could provide the systems with capabilities to perform artillery missions similar to or greater than the Crusader include a multi- role armament system. This possible system could feature a 105-mm cannon that may have a non-line-of-sight capability out to a range of about 50 kilometers. Also, the Army is considering an advanced missile system that could be comprised of small-containerized missiles, known as NetFires, which are projected to have a range of 50 to 100 kilometers. A high-level Army official told us that he believes, based on recent technical briefings, that the initial version of the Future Combat Systems will not have the capabilities to meet the same artillery missions as the Crusader.

Conclusions

Moving into product development without demonstrating critical technologies in an operational environment increases the risk of cost overruns, schedule delays, and performance shortfalls. As currently planned, the majority of the critical Crusader technologies will have been demonstrated in a relevant environment but not the important operational environment. If the Crusader program follows the approach of moving into product development with less mature technologies, the program will need to continue to develop and demonstrate those technologies while concentrating on integrating subsystems into the system, testing at the subsystem and system levels, and preparing for production. As a result, technical problems, if they occur, will need to be resolved in the higher cost environment of system development and demonstration. On the other hand, demonstrating the critical technologies in an operational environment before entering system development and demonstration could necessitate more time and money than currently planned before the milestone B decision, but such investments would be relatively small compared to solving technical problems after the decision.

The Army restructured the Crusader program to improve the system’s strategic deployability by reducing the system’s weight. The lighter-weight system, however, may not provide a significant improvement to strategic deployability. At this time, the Army is making design trade-offs to meet its weight requirement and it is not clear whether the Army can maintain its lighter weight goals throughout the development, production, and fielding of the Crusader system. Given the uncertainty, the Army risks making unnecessary cost, schedule, and performance trade-offs to meet deployability requirements that may not be clearly justified.

The Army has not ruled out the possibility that it will field the Future Combat Systems with the ability to meet the same artillery mission as the Crusader in the same year the Crusader is fielded. However, the extent of this apparent overlap will not be clear until the potential capabilities and schedule of the initial version of the Future Combat Systems are determined. Therefore, it is important that the Army ensure that the projected capabilities and schedule for the initial Future Combat Systems are considered in the Crusader milestone decision.

Recommendations for Executive Action

To reduce the risk of schedule delays and increased costs in the product development phase of the Crusader program, we recommend that the secretary of defense direct the secretary of the army to dedicate the resources necessary to ensure that the critical Crusader technologies are demonstrated, at the component and subsystem level, in an operational environment before the program commits to product development at milestone B.

To confirm the value and usefulness of the Crusader program’s deployability requirement, we recommend that the secretary of defense direct the secretary of the army to conduct an analysis, before the decision to enter product development, to determine how important it is to deploy two Crusaders howitzers on a single C-17 aircraft. If it is important to the Army, we recommend that the secretary of defense direct the secretary of the army to establish, as a key performance parameter, the maximum per vehicle weight that would allow the C-17 aircraft to carry two Crusader howitzers. If the analysis determines that the redesigned Crusader does not significantly improve the system’s military utility, we recommend that the secretary of defense direct the secretary of the army to reduce the priority placed on attaining the 42-ton weight limit.

Finally, to ensure the Army does not invest in two weapon systems that will meet the same artillery missions at the same time, we recommend that the secretary of defense direct the secretary of the army to determine, based on available data, the potential capabilities and schedule of the initial version of the Future Combat Systems and the implication of those capabilities and schedule on the Crusader’s utility to the Army before making the decision on beginning the Crusader’s system development and demonstration—currently scheduled for April 2003.

Agency Comments and Our Evaluation

In written comments on a draft of this report, the director of strategic and tactical systems, within the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, said that DOD did not agree with our recommendation that the Crusader technologies be demonstrated in an operational environment before the program commits to product development. DOD said that the Crusader program was a simulation-based acquisition program and, as such, evaluates system, component, and subsystem performance and technology readiness using modeling and simulation validated with test stands, integration laboratories, and subsystem prototypes. DOD questioned our definition of critical Crusader technologies and said that the track, for example, was selected by us as a critical technology and assessed as a TRL 5 despite the Army’s many years of expertise in track development. DOD also said that the Crusader is currently demonstrating performance equal to or in excess of threshold requirements for the final system. Finally, DOD said that changing the Crusader’s acquisition strategy to accommodate building system level prototypes required to demonstrate TRL 7 for all critical technologies would add significantly to the development time and expense without significantly reducing risk or improving performance. The full text of DOD’s comments is included in appendix II.

We agree that modeling and simulation is a key and accepted practice in any modern development program. However, we have found that programs need to demonstrate a high level of technology maturity before committing to product development. As shown by our past reviews, the best practice standard is that technology must be demonstrated, at the component or subsystem level, in an operational environment to be considered mature enough for entering product development. We believe that a program should use this best practice to assure success in meeting its cost and schedule goals.

The determination of the critical Crusader technologies was a joint effort between the Crusader program office and us. We defined critical Crusader technologies as those required to meet the Crusader’s key performance parameters and developed the initial critical technology list. Crusader program office engineers reviewed the initial list, suggested revisions, and agreed that the revised critical technologies list was complete and appropriate. Also, our analysts and program office engineers jointly arrived at the appropriate TRL for each critical technology. In addition, DOD’s statement that track should not be a critical Crusader technology or should have been assessed at a higher TRL because of the Army’s many years of expertise in track development underscores the value of the TRL methodology. Track was included as a critical Crusader technology because the Crusader cannot meet its mobility key performance parameter without track. The track was assessed at TRL 5 because the Crusader program was developing a new lighter-weight track. The Army plans to demonstrate it in an operational environment after milestone B. TRLs measure whether sufficient knowledge has been accumulated with respect to each application of a technology, not the development difficulty of the technology or whether the technology has been previously used in another application. The issue is not whether a technology like the newly developed track will ever work, but how much time and effort will be needed to demonstrate its maturity in this application. The Crusader system development and demonstration phase does not have much time between prototype testing and procurement of long-lead items for production to adjust for any delays or problems in prototype testing caused by technology problems. Such delays or problems could either delay the long- lead item procurement or reduce the amount of information available when committing to the procurement.

DOD’s assessment that the Crusader system is currently demonstrating performance equal to or in excess of threshold requirements for the final system is based mainly on modeling, simulations, and laboratory tests because the program has not produced the final system. As mentioned above, best practice calls for critical technologies to be demonstrated in an operational environment not in models, simulations, or laboratory environments before entering product development.

DOD stated that building the full system prototype required to demonstrate TRL 7 would add significant time and expense to the program. However, demonstrating at TRL 7 does not require a full system prototype but only a prototype of the component or subsystem that contains a new technology. The demonstration can be accomplished by putting the new component or subsystem, such as an engine, on a surrogate vehicle; that is, a vehicle that already exists. The report’s point is that using full system prototypes to demonstrate the maturity of critical technologies during the product development phase, as planned in the Crusader program, is potentially more costly than using component or subsystem prototypes to do so during the technology development phase. Problems that occur during required demonstrations may cause program delays in either phase, but as noted in the report, the delay is more expensive during the product development phase.

DOD stated that it partially agreed with our recommendation to conduct an analysis to determine the importance of the deployability requirement and said that the current requirement is not considered a key performance parameter and, as a result, the Army is allowed to make trade-offs between the requirement and system cost and performance. DOD further stated that the Army plans to review the Crusader's requirements prior to the 2003 milestone B decision as required by regulations. We believe that an analysis to determine the importance of deploying two Crusader howitzers on a C- 17 aircraft should be conducted as soon as possible to provide the Army greater flexibility and knowledge in considering its ongoing trade-off decisions needed to meet weight requirements.

DOD stated that it partially agreed with our recommendation to determine the potential capabilities and schedule of the initial version of the Future Combat Systems before making the decision to begin Crusader product development and stated that the Crusader’s capabilities are intended to complement rather than compete with or be redundant to the capabilities of the Future Combat Systems. We continue to believe that DOD cannot determine whether the two systems will be complementary or redundant without knowledge of the initial Future Combat Systems capabilities and fielding schedule. DOD does not have this knowledge. We continue to believe that this knowledge needs to be considered as part of the decision to allow the Crusader program to enter product development. We have rewritten the recommendation to clarify its intent.

Scope and Methodology

To determine the readiness of the Crusader program to enter the system development and demonstration phase, we assessed, along with engineers from the Crusader Project Office, the current maturity of the critical Crusader technologies using the technology readiness level tool. We identified the Crusader technologies we believed were critical to meeting the Crusader system key performance parameters. Program engineers reviewed our list, suggested revisions, and agreed that the revised critical technologies list was complete and appropriate. After considering the program’s plans for maturing the critical technologies before milestone B, we jointly determined the probable TRL levels of each of the critical technologies at the milestone. This determination assumed that the program office would successfully execute its existing plans for demonstrating some of the technologies before the milestone.

To assess the status of the Crusader software development, we used project management criteria derived from Software Engineering Institute's Software Development Capability Maturity Model. We visited the Crusader prime contractor, met with Army and contractor officials, observed software development and test facilities, and examined project information. We also obtained and reviewed project documentation from the prime contractor and the Army program office.

To assess the Crusader program’s ability to meet the Crusader reduced weight requirements and improve the Crusader system’s strategic deployability, we analyzed the Army’s plans and requirements for reducing the weight of the Crusader and requested that the Army perform an analysis of the improvement in strategic deployability that the reduced weight Crusader system would provide compared to the original weight Crusader system. For this analysis, at our request, the Army determined the number of Crusader systems to be deployed, the other equipment and supplies that were required to be deployed with the Crusader systems, and the range of the aircraft used for the deployment. We reviewed the results of the Army’s Crusader deployment analysis.

To determine whether the Army is developing the Crusader and the Future Combat Systems to be fielded at the same time and to meet the same artillery missions, we analyzed and compared the Crusader and Future Combat Systems schedules and reviewed the Crusader system operational requirements documents. The Future Combat Systems do not have operational requirements documents at this stage of development. Also, we discussed with appropriate officials in the Army’s Objective Force Task Force, the Army’s artillery school, and the Crusader and the Future Combat Systems programs (1) the probability that the two programs would meet their individual schedules and (2) the potential technologies that might be used in the Future Combat Systems to provide it with artillery capabilities.

In performing our work, we obtained documents and interviewed officials involved in the Crusader and the Future Combat Systems programs in the Office of the Deputy Chief of Staff for Operations, Washington, D.C.; U.S. Army Training and Doctrine Command, Fort Monroe, Virginia; U.S. Army Field Artillery School and Center, Fort Sill, Oklahoma; the Defense Advanced Research Projects Agency, Arlington, Virginia; the Military Traffic Management Command, Newport News, Virginia; the U.S. Air Force, Air Mobility Command, St. Louis, Missouri; the U.S. Air Force Aeronautical Systems Command, Dayton, Ohio; the Crusader Project Office, Picatinny Arsenal, New Jersey; and the prime contractor’s Minneapolis, Minnesota, facility.

We conducted our review between March 2001 and October 2001 in accordance with generally accepted government auditing standards.

We also are sending copies of this report to the appropriate congressional committees; the director, Office of Management and Budget; and the secretaries of defense and the army. We will also provide copies to others upon request.

If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or William R. Graveline at (256) 650-1414. Key contributors are listed in appendix III.

Technology Readiness Levels and Their Descriptions

1. Basic principles observed and reported.

Lowest level of technology readiness. Scientific research begins to be translated into technology’s basic properties.

Invention begins. Once basic principles are observed, practical applications can be invented. The application is speculative and there is no proof or detailed analysis to support the assumptions. Examples are still limited to paper studies.

Active research and development is initiated. This includes analytical studies and laboratory studies to physically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative.

4. Component and/or breadboard validation in laboratory environment.

Basic technological components are integrated to establish that the pieces will work together. This is relatively “low fidelity” compared to the eventual system. Examples include integration of “ad hoc” hardware in a laboratory. 5. Component and /or breadboard validation Fidelity of breadboard technology increases significantly. The basic technological in relevant environment. components are integrated with reasonably realistic supporting elements so that the technology can be tested in a simulated environment. Examples include “high fidelity” laboratory integration of components.

Representative model or prototype system, which is well beyond the breadboard tested for level 5, is tested in a relevant environment. Represents a major step up in technology’s demonstrated readiness. Examples include testing a prototype in a high fidelity laboratory environment or in a simulated operational environment.

Prototype near or at planned operational system. Represents a major step up from level 6, requiring the demonstration of an actual system prototype in an operational environment. Examples include testing the prototype in a test bed aircraft.

Technology has been proven to work in its final form and under expected conditions. In almost all cases, this level represents the end of true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications. 9. Actual system proven through successful Actual application of technology in its final form and under mission conditions, such as mission operations. those encountered in operational test and evaluation. Examples include using the system under operational mission conditions.

Comments from the Department of Defense

GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to those named above, the following individuals made significant contributions to this report: Robert L. Ackley; Nabajyoti Barkakati; Paul L. Francis; Lawrence D. Gaston, Jr.; Matthew B. Lea; Gary L. Middleton; Madhav S. Panwar; Robert J. Stolba; and John P. Swain.

GAO’s Mission

The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO E-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily e-mail alert for newly released products” under the GAO Reports heading.

Order by Mail or Phone

Visit GAO’s Document Distribution Center

To Report Fraud, Waste, and Abuse in Federal Programs

Web site: www.gao.gov/fraudnet/fraudnet.htm, E-mail: fraudnet@gao.gov, or 1-800-424-5454 or (202) 512-7470 (automated answering system).

Public Affairs