DOD's Standard Procurement System: Continued Investment Has Yet to Be Justified
Released: 2002-02-07T00:00:00

Letter

SPS: A Brief Description and History

In November 1994, the Office of the Director of Defense Procurement initiated the SPS program to acquire and deploy a single automated system to perform all contract-management-related functions for all DOD organizations. At that time, life-cycle costs were estimated to be about $3 billion over a 10-year period.

From 1994 to 1996, the department defined SPS requirements and solicited commercially available vendor products for satisfying these requirements. Subsequently, in April 1997, the department awarded a contract to American Management Systems (AMS), Incorporated, to (1) use AMS’s commercially available contract management system as the foundation for SPS, (2) modify this commercial product as necessary to meet DOD requirements, and (3) perform related services. The department also directed the contractor to deliver functionality for the system in four incremental releases. The department later increased the number of releases across which this functionality would be delivered to seven, reduced the size of the increments, and allowed certain more critical functionality to be delivered sooner (see table 1 for proposed SPS functionality by increment).

Since our report of July 2001, DOD has revised its plans. According to the SPS program manager, current plans no longer include increments 6 and 7 or releases 5.0 and 5.1. Instead, release 4.2 (increment 5) will include at least three, but not more than seven, subreleases. At this time, only the first of the potentially seven 4.2 subreleases is under contract. This subrelease is scheduled for delivery in April 2002, with deployment to the Army and the Defense Logistics Agency scheduled for June 2002. Based on the original delivery date, release 4.2 is about one year overdue.

The department reports that it has yet to define the requirements to be included within the remaining 4.2 subreleases, and has not executed any contract task orders for these subreleases. According to SPS officials, they will decide later this year whether to invest in these additional releases.

As of December 2001, the department reported that it had deployed four SPS releases to over 777 locations. The Director of Defense Procurement (DDP) has responsibility for the SPS program, and the CIO is the milestone decision authority for SPS because the program is classified as a major Defense acquisition.

Numerous SPS Concerns Have Been Raised by Us and Others

Our July 2001 report detailed program problems and investment management weaknesses. To address these weaknesses, we recommended, among other things, that the department report on the lessons to be learned from its SPS experience for the benefit of future system acquisitions. Similarly, other reviews of the program commissioned by the department in the wake of our review raised similar concerns and identified other problems and management weaknesses. The findings from our report are summarized below in two major categories: lack of economic justification for the program and inability to meet program commitments. We also summarize the findings of the other studies.

DOD Had Not Economically Justified Its Investment in SPS

The Clinger-Cohen Act of 1996, OMB guidance, DOD policy, and practices of leading organizations provide an effective framework for managing information technology investments, not just when a program is initiated, but continuously throughout the life of the program. Together, they provide for (1) economically justifying proposed projects on the basis of reliable analyses of expected life-cycle costs, benefits, and risks; and (2) using these analyses throughout a project’s life-cycle as the basis for investment selection, control, and evaluation decisionmaking, and doing so for large projects (to the maximum extent practical) by dividing them into a series of smaller, incremental subprojects or releases and individually justifying investment in each separate increment on the basis of costs, benefits, and risks.

The department had not met these investment management tenets for SPS. First, the latest economic analysis for the program—dated January 2000— was not based on reliable estimates because most of the cost estimates in the 2000 economic analysis were estimates carried forward from the April 1997 analysis (adjusted for inflation). Only the cost estimates being funded and managed by the SPS program office, which were 13 percent of the total estimated life-cycle cost in the analysis, were updated in 2000 to reflect more current contract estimates and actual expenditures/ obligations for fiscal years 1995 through 1999. Moreover, the military services, which share funding responsibility with the SPS program office for implementing the program, questioned the reliability of these cost estimates. However, this uncertainty was not reflected in the economic analysis using any type of sensitivity analysis. A sensitivity analysis would have disclosed for decisionmakers the investment risk being assumed by relying on the estimates presented in the economic analysis.

Moreover, the latest economic analysis (January 2000) was outdated because it did not reflect the program’s current status and known problems and risks. For instance, this analysis was based on a program scope and associated costs and benefits that anticipated four software releases. However, as mentioned previously, the program now consists of five releases, and subreleases within releases, in order to accommodate changes in SPS requirements. Estimates of the full costs, benefits, and risks relating to this additional release and its subreleases were not part of the 2000 economic analysis. Also, this analysis did not fully recognize actual and expected delays in meeting SPS’s full operational capability milestone, which had been slipped by 3½ years and DOD officials say that further delays are currently expected. Such delays not only increase the system acquisition costs but also postpone, and thus reduce, accrual of system benefits. Further, several DOD components are now questioning whether they will even deploy the software, which would further reduce SPS’s cost effectiveness calculations in the 2000 economic analysis.

Second, the department had not used these analyses as the basis for deciding whether to continue to invest in the program. The latest economic analysis showed that SPS was not a cost-beneficial investment because the estimated benefits to be realized did not exceed estimated program costs. In fact, the 2000 analysis showed estimated costs of $3.7 billion and estimated benefits of $1.4 billion, which was a recovery of only 37 percent of costs. According to the former SPS program manager, this analysis was not used to manage the program and there was no DOD requirement for updating an economic analysis when changes to the program occurred.

Third, DOD had not made its investment decisions incrementally as required by the Clinger-Cohen Act and OMB guidance. That is, although the department is planning to acquire and implement SPS as a series of five increments, it has not made decisions about whether to invest in each release on the basis of the release’s expected return on investment, as well as whether prior releases were actually achieving return-on-investment expectations. In fact, for the four increments that have been deployed, the department had not validated whether the increments were providing promised benefits and was not accounting for the costs associated with each increment so that it could even determine actual return on investment.

Instead, the department had treated investment in this program as one, monolithic investment decision, justified by a single, “all-or-nothing” economic analysis. Our work has shown that it is difficult to estimate, with any degree of accuracy, cost and schedule estimates for many increments to be delivered over many years because later increments are not well understood or defined. Also, these estimates are subject to change based on actual program experiences and changing requirements. This “all-or- nothing” approach to investing in large system acquisitions, like SPS, has repeatedly proven to be ineffective across the federal government, resulting in huge sums being invested in systems that do not provide commensurate benefits.

DOD Had Not Met or Did Not Know if It Had Met SPS Commitments

Measuring progress against program commitments is closely aligned with economically justifying information-technology investments, and is equally important to ensuring effective investment management. The Clinger- Cohen Act, OMB guidance, DOD policy, and practices of leading organizations provide for making and using such measurements as part of informed investment decisionmaking.

DOD had not met key commitments and was uncertain whether it was meeting other commitments because it was not measuring them. (See table 2 for a summary of the department’s progress against commitments.) two analyses, such as the number and dollar value of estimated benefits, and the information gathered did not map to the 22 benefit types listed in the 1997 economic analysis. Instead, the study collected subjective judgments (perceptions) that were not based on predefined performance metrics for SPS capabilities and impacts. Thus, the department was not measuring SPS against its promised benefits.

The former program manager told us that knowing whether SPS was producing value and meeting commitments was not the program office’s objective because there was no departmental requirement to do so. Rather, the objective was simply to acquire and deploy the system. Similarly, CIO officials told us that the department was not validating whether deployed releases of SPS were producing benefits because there was no DOD requirement to do so and no metrics had been defined for such validation. However, the Clinger-Cohen Act of 1996 and OMB guidance emphasize the need to have investment management processes and information to help ensure that information-technology projects are being implemented at acceptable costs and within reasonable and expected time frames and that they are contributing to tangible, observable improvements in mission performance (i.e., that projects are meeting the cost, schedule, and performance commitments upon which their approval was justified). For programs such as SPS, DOD required this cost, schedule, and performance information to be reported quarterly to ensure that programs did not deviate significantly from expectations. In effect, these requirements and guidance recognize that one cannot manage what one cannot measure.

Other Studies Reported Similar Findings and Identified Other Concerns

Shortly after receiving our draft report for comment, the department initiated several studies to determine the program’s current status, assess program risks, and identify actions to improve the program. These studies focused on such areas as program costs and benefits, planned commitments, requirements management, program office structure, and systems acceptance testing. Consistent with our findings and recommendations, these studies identified the need to establish performance metrics that will enable the department to measure the program’s performance and tie these metrics to benefits and customer satisfaction; clearly define organizational accountability for the program; provide training for all new software releases; standardize the underlying business processes and rules that the system is to support; acquire the software source code; and address open customer concerns to ensure user satisfaction.

In addition, the department found other program management concerns not directly within the scope of our review, such as the need to appropriately staff the program management office with sufficient resources and address the current lack of technical expertise in areas such as contracting, software engineering, testing, and configuration management; modify the existing contract to recognize that the system does not employ a commercial-off-the-shelf software product, but rather is based on customized software product; establish DOD-controlled requirements management and acceptance testing processes and practices that are rigorous and disciplined; and assess the continued viability of the existing contractor.

DOD Has Begun Addressing Problems, But SPS’s Future Remains Uncertain

To address the many weaknesses in the SPS program, we made several recommendations in our July 2001 report. Specifically, we recommended that (1) investment in future releases or major enhancements to the system be made conditional on the department first demonstrating that the system is producing benefits that exceed costs; (2) future investment decisions, including those regarding operations and maintenance, be based on complete and reliable economic justifications; (3) any analysis produced to justify further investment in the program be validated by the Director, Program Analysis and Evaluation; (4) the Assistant Secretary of Defense for Command, Control, Communications, and Intelligence (C3I) clarify organizational accountability and responsibility for measuring SPS program against commitments and to ensure that these responsibilities are met; (5) program officials take the necessary actions to determine the current state of progress against program commitments; and (6) the Assistant Secretary of Defense for C3I report by October 31, 2001, to the Secretary of Defense and to DOD’s relevant congressional committees on lessons learned from the SPS investment management experience, including what actions will be taken to prevent a recurrence of this experience on other system acquisition programs.

DOD’s reaction to our report was mixed. In official comments on a draft of our report, the Deputy CIO generally disagreed with our recommendations, noting that they would delay development and deployment of SPS. Since that time, however, the department has acknowledged its SPS problems and begun taking steps to address some of them. In particular, it has done the following.

The department has established and communicated to applicable DOD organizations the program’s chain-of-command and defined each participating organization’s responsibilities. For example, the Joint Requirements Board was delegated the responsibility for working with the program users to define and reach agreement on the needed functionality for each software release.

The department has restructured the program office and assigned additional staff, including individuals with expertise in the areas of contracting, software engineering, configuration management, and testing. However, according to the current program manager, additional critical resources are needed, such as two computer information technology specialists and three contracting experts.

It has renegotiated certain contract provisions to assume greater responsibility and accountability for the requirements management and testing activities. For example, DOD, rather than the contractor, is now responsible for writing the test plans. However, additional contract changes remain to be addressed, such as training, help-desk structure, facilities support, and system operations and maintenance.

The department has designated a user-satisfaction manager for the program and defined forums and approaches intended to better engage users.

It has established a new testing process, whereby program officials now develop the test plans and maintain control over all software testing performed.

In addition, SPS officials have stated their intention to prepare analyses for future program activities beyond those already under contract, such as the acquisition of additional system releases, and use these analyses in deciding whether to continue to deploy SPS or pursue another alternative; define system performance metrics and use these metrics to assess the extent to which benefits have been realized from already deployed system releases; and report on lessons learned from its SPS experience to the Secretary of Defense and relevant congressional committees.

The department’s actions and intentions are positive steps and consistent with our recommendations. However, much remains to be accomplished. In particular, the department has yet to implement our recommendations aimed at ensuring that (1) future releases or major enhancements to the system be made conditional on first demonstrating that the system is producing benefits that exceed costs and (2) future investment decisions, including those regarding operations and maintenance, be based on a complete and reliable economic justification.

We also remain concerned about the future of SPS for several additional reasons. First, definitive plans for how and when to justify future system releases or major enhancements to existing releases do not yet exist. Second, SPS officials told us that release 4.2, which is currently under contract, may be expanded to include functionality that was envisioned for releases 5.0 and 5.1. Including such additional functionality could compound existing problems and increase program costs. Third, not all defense components have agreed to adopt SPS. For example, the Air Force has not committed to deploying the software; the National Imagery and Mapping Agency, the Defense Advanced Research Projects Agency, and the Defense Intelligence Agency have not yet decided to use SPS; and the DOD Education Agency has already adopted another system because it deemed SPS too expensive.