DOD Systems Modernization: Continued Investment in the Standard Procurement System Has Not Been Justified
Released: 2001-09-04T00:00:00

Letter

Background

DOD’s procurement process spans numerous Defense agencies and military services. This process provides for acquiring supplies and services from nonfederal sources and, when necessary, administering the related contractual instruments. It also provides for administering grants, cooperative agreements, and other transactions executed by contracting offices. The procurement process begins with the receipt of a requirement and ends at the contract closeout. (See fig. 1 for a simplified diagram of the procurement process, the interaction of this process with the logistics and financial management processes, and those functions within the procurement process that SPS is to support.)

SPS: A Brief Description

In November 1994, DOD’s Director of Defense Procurement (DDP) initiated the SPS program to acquire and deploy a single automated system to perform all contract management-related functions within DOD’s procurement process for all DOD organizations and activities. From 1994 to 1996, DOD defined SPS requirements and solicited commercially available vendor products for satisfying these requirements. DOD subsequently awarded a contract to American Management Systems (AMS), Incorporated, in April 1997, to (1) use its commercially available contract management system as the foundation for SPS, (2) modify this commercial product as necessary to meet the requirements, and (3) perform related services. DOD also directed the contractor to deliver SPS functionality in four incremental releases. The department later increased the number of releases across which this functionality would be delivered to seven; reduced the size of the increments; and allowed certain, more critical functionality to be delivered sooner.

Over the last 4 years, DOD and AMS have deployed four releases to 773 locations in support of 21,900 users. The fifth release was delivered in February 2001 for acceptance testing; however, due to software deficiencies, this release was sent back to the vendor for rework and has not been deployed. AMS is expected to provide a second version of this release to DOD in July 2001 for additional testing. If accepted, the fifth release is to be deployed to about 4,500 users beginning in fiscal year 2002. DOD has not yet contracted for the sixth and seventh releases. (See table 2 for the status of the various software releases, and table 3 for the summary of SPS functionality by increment.)

As planned, SPS is to be used to prepare contracts and contract-related documents and to support contracting staff in monitoring and administering them. SPS also is intended to standardize procurement business practices and data elements throughout DOD and to provide timely, accurate, and integrated contract information. Using SPS, the goal is that required contract and contract payment data will be entered once— at the source of the data’s creation—and be stored in a single database. As depicted in figure 1, SPS is to electronically interface with DOD’s logistics community, which is the source of goods and services requests, and with the Defense Finance and Accounting Service (DFAS), which is responsible for contract payments.

DDP is organizationally within the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics. However, as shown in table 3, the management responsibility for SPS is shared among several organizations.

Previous Reviews of SPS Have Identified Management Concerns

Since 1996, DOD’s Office of the Inspector General (OIG) has issued three reports critical of SPS. In September 1996, the OIG reported that the needs of SPS users might not be met and that actual costs could exceed proposed costs because, among other things, the functional requirements were very broad, existing commercial software required substantial modification, and adequate development and operational test strategies had not been developed.

The OIG later reported in May 1999 that SPS lacked critical functionality and concluded that the system may not meet mission needs with regard to standardizing procurement policy, processes, and procedures. The report also noted that users were receiving inadequate system training, guidance, and support, thereby forcing users to develop inefficient system workarounds. Finally, the report raised concerns about the cost- effectiveness of DOD’s contractual reliance on a single vendor to provide system support over the life of SPS, adding that an expanded license was needed to give DOD the ability to competitively compete support services.

In March 2001, the OIG reported that lack of system functionality was still a serious program concern, productivity had not increased with the implementation of version 4.1, and users were generally dissatisfied with SPS.

SPS program officials generally concurred with the OIG’s findings and agreed to issue guidance on the acquisition of commercial software for major automated information systems, support development of accurate life-cycle cost estimates for SPS, clarify responsibilities for the program office, the contractor, and the evaluate the cost and benefits of obtaining additional license rights and renegotiating the contract, require the program office to be aware of additional support contracts, and suggest that the component organizations provide funds to the program office to better integrate user needs, better coordinate training needs among the DOD component organizations, and require that before any future deployments of SPS, the DOD component organizations determine that the version meets their functional requirements and to identify the number of licenses required.

Also, in response to the OIG’s March 2001 report, the SPS program office initiated its own study in June 2000 to assess the extent to which benefits will be realized as a result of its implementation of version 4.1 of SPS. The program office plans to publish the study results by October 2001.

DOD Either Is Not Meeting or Does Not Know If It Is Meeting SPS Commitments

Federal information technology (IT) investment management requirements and guidance recognize the need to measure investment programs’ progress against commitments. In the case of SPS, DOD is not meeting key commitments and is not measuring whether it is meeting other commitments. According to the program manager, the program office is not responsible for ensuring that all program commitments are being met. Rather, the program office’s sole task is to acquire and deploy an SPS system solution that meets defined functional requirements. Given that SPS is a major Defense acquisition, the DOD CIO is the decisionmaking authority for SPS. However, according to officials in the CIO’s office, SPS has continued to be approved and funded regardless of progress against expectations on the basis of decisions made by individuals organizationally above the CIO’s office. Without measuring and reporting progress against program commitments and taking the appropriate actions to address significant deviations, DOD runs the serious risk of investing billions of dollars in a system that will not produce commensurate value.

Measuring Progress Against Program Commitments Is a Key to Effective Investment Management

The Clinger-Cohen Act of 1996 and Office of Management and Budget (OMB) guidance emphasize the need to have investment management processes and information to help ensure that IT projects are being implemented at acceptable costs and within reasonable and expected time frames and that they are contributing to tangible, observable improvements in mission performance (i.e., that projects are meeting the cost, schedule, and performance commitments upon which their approval was justified). For programs such as SPS, DOD requires this cost, schedule, and performance information to be reported quarterly to ensure that programs do not deviate significantly from expectations. In effect, these requirements and guidance recognize that one cannot manage what one cannot measure.

DOD Has Not Met Key SPS Program Commitments

DOD has not met key SPS commitments concerning the timing of product delivery, user satisfaction with system performance, and the use of a commercial system solution, as discussed below: DOD committed to SPS’ being fully operational at all sites by March 31, 2000; however, this date has slipped by 3-½ years and is likely to slip further.

Currently, DOD has established a September 30, 2003, milestone for making SPS fully operational, and the program manager attributed this delay to (1) problems encountered in modifying and testing the contractor’s commercial product to meet DOD’s requirements and (2) an increase in requirements. However, the SPS Joint Requirements Board chairperson stated that no additional requirements have been approved. Instead, the original requirements were clarified for the contractor to better ensure that the needs of the user would be met.

However, satisfying even this revised commitment will be problematic for several reasons. First, the 2003 milestone does not recognize DOD components’ testing activities that need to occur before the system could be fully operational. For example, Department of the Air Force officials told us that they are typically 6 to 12 months behind the program office’s deployment milestones because of additional testing that the Air Force performs before it implements the software releases. Second, the 2003 milestone has not been updated to reflect the impact of events. For example, version 4.1, the latest deployed release, was recently changed from a single release to five subreleases to correct software problems discovered during operation of version 4.1; and version 4.2 recently failed acceptance testing, and the vendor is still attempting to correct identified defects. Third, the official responsible for SPS independent operational test and evaluation, as well as the official in DOD’s Office of Program Analysis and Evaluation who is responsible for reviewing the SPS economic analyses, told us that this milestone is likely to slip further. The reasons that these officials cited included incomplete system functionality, increased system complexity, and inadequate training.

DOD committed to SPS’ satisfying the needs of its contracting community and meeting specified system requirements, ultimately increasing contracting efficiency and effectiveness. However, according to a recent DOD OIG report, approximately 60 percent of the user population surveyed was not satisfied with the system’s functionality and performance, resulting in the continued use of legacy systems and/or manual processes in lieu of SPS.

Similarly, another DOD report describes SPS as unstable because the system frequently goes down, meaning that it is unexpectedly unavailable to users who are logged on and using the system, which, in turn, causes users to lose information. The report also notes that users complained that previously identified problems were not being resolved in later software releases, and that requested changes or enhancements were not being made. According to the program manager, at any one time, there was a backlog of 100 to 200 problems that needed to be addressed in order for SPS to meet specified requirements. In light of these challenges in meeting requirements and satisfying user needs, the official responsible for independent operational test and evaluation of SPS said that DOD should not invest in additional releases beyond version 4.2.

In delivering SPS, DOD was to use a commercially available software product. However, the contractor has modified the commercial product extensively in an attempt to satisfy DOD’s needs; thus, SPS is now a DOD-unique system solution.

According to the program manager, DOD knew when it selected the commercial product that the product provided only 45 percent of the functionality that DOD needed, and that extensive new software development and existing software modification were necessary. Nevertheless, the product was chosen because no commercial product was available that met DOD’s requirements, and, of the products available, DOD believed that AMS’ product and company would provide the best value.

In accordance with industry best practices, software modifications to a commercial product should not exceed 10 to 15 percent. Beyond this degree of software change, experts generally consider development or acquisition of a custom system solution more cost-effective. Further, DOD guidance states that custom modifications to a commercial item, even if made and implemented by the commercial item’s vendor, result in custom system solutions. This guidance emphasizes the use of commercial items to reduce life-cycle costs and increase product reliability and availability. Since SPS is not a commercial product, DOD will not be able to take advantage of the reduced cost and risk associated with using proven technology that is used by a wide customer base.

DOD Does Not Know What Progress Is Being Made Against Other SPS Commitments

When it began the program, DOD promised that SPS would produce such benefits as (1) replacing 76 legacy systems and manual processes with a single system and thereby reducing procurement system operations and maintenance costs by an unspecified amount, (2) standardizing policies, processes, and procedures across the Department, and (3) reducing problem disbursements. However, DOD does not know the extent to which SPS is meeting each of these expectations, even though versions have been deployed to about 773 user locations.

First, although DOD reports that it has retired two major legacy systems, neither the program office nor the DOD CIO office could provide us with information on what, if any, savings have been realized by doing so. Additionally, program officials told us that the number of legacy systems and manual processes that SPS is to replace is now significantly less than the 76 originally used to justify the program. In response to our inquiry, the SPS program manager recently surveyed the DOD component organizations to determine the number of legacy systems. According to the results of the survey, there were 55 legacy procurement systems. See table 4 for the status of these systems as of June 2001. According to the SPS program manager, 45 of the 55 systems remain, and 10 to 12 of these systems are to be replaced by SPS. However, another program official noted that SPS was always intended to replace only 14 major legacy systems. In either case, the latest economic analysis has not been updated to reflect this change in the number of systems to be replaced, and the associated cost savings are not known.

Second, the standardization of policies, processes, and procedures benefit is not materializing because each military service is either in the process of developing, or has plans to develop, its own unique policies, processes, and procedures.

Third, program officials were unable to provide evidence that implementing SPS has reduced problem disbursements or achieved the benefits outlined in the economic analysis. In fact, the latest economic analysis no longer even cites reducing problem disbursements as a benefit because the DOD components’ position was that SPS would not completely address this problem. According to the program manager and CIO officials, there is no DOD policy that requires them to assess whether the expected benefits are in fact being realized.

When the SPS program began, DOD also committed to a system life-cycle cost of about $3 billion over a 10-year period. However, total actual program costs are not being accumulated and monitored against estimates, which in 2000 were revised to about $3.7 billion (a 28-percent increase). Thus, DOD does not know what has been spent on the program by all DOD component organizations. To date, the only actual program costs being collected and reported are those incurred by the SPS program office, which DOD reports to be about $322 million through September 30, 2000.

To determine the total cost of the SPS program through September 30, 2000, we requested cost information from 18 Defense agencies and the four military services. These DOD components reported that they have collectively spent approximately $125 million through September 30, 2000. However, these reported costs are not complete because (1) 4 of the 22 DOD components did not respond, (2) components reported that SPS costs were being captured with other programs and could not be allocated accurately, and (3) all SPS costs, such as employee salaries and system infrastructure costs, were not included. According to program officials, no single DOD organization is responsible for accumulating the full DOD cost of SPS.

Without knowing the extent to which SPS is meeting cost-and-benefit expectations, DOD is not in a position to make informed, and thus justified, decisions on whether and how to proceed further on the program. Such a situation introduces a serious risk of investing in a system that will not produce a positive net present value (i.e., estimated benefits to be realized would exceed estimated program costs).

DOD Has Not Economically Justified Investing in SPS

Federal IT investment management requirements and guidance, as well as DOD policy, recognize the need to economically justify IT projects before investing in them and to justify them in an incremental manner in an effort to spread the risk of doing many things over many years on large projects across smaller, more manageable subprojects. However, the department has not economically justified investing in SPS because its own analysis shows that expected life-cycle benefits are less than estimated life-cycle costs. Moreover, DOD is not approaching its investment in SPS on an incremental basis. Nevertheless, DOD continues to invest hundreds of millions of dollars in SPS each year, running the serious risk of spending large sums of money on a system that does not produce commensurate value. According to program and CIO officials, DOD continues to invest these funds because individuals above the CIO’s office decided that SPS was a departmental priority.

Reliable Economic Analysis and Incremental Investment Are Tenets of Effective IT Investment Management

The Clinger-Cohen Act of 1996 and OMB guidance provide an effective framework for IT investment management. Together, they set requirements for (1) economically justifying proposed projects on the basis of reliable analyses of expected life-cycle costs, benefits, and risks, (2) using these analyses throughout a project’s life cycle as the basis for investment selection, control, and evaluation decisionmaking, and (3) doing so for large projects (to the maximum extent practical) by dividing them into a series of smaller, incremental subprojects or releases. By doing so, the tremendous risk associated with investing large sums of money over many years in anticipation of delivering capabilities and expected business value far into the future can be spread across project parts that are smaller, of a shorter duration, and capable of being more reliably justified and more effectively measured against cost, schedule, capability, and benefit expectations.

DOD policy also reflects these investment principles by requiring that investments be justified by an economic analysis and, more recently, that investment decisions for major programs, like SPS, be made incrementally by ensuring that each incremental part of the program delivers measurable benefit, independent of future increments. According to the policy, the economic analysis is to reflect both life-cycle cost and benefits estimates, including a return-on-investment calculation, to demonstrate that a proposal to invest in a new system is economically justified before that investment is made.

SPS Economic Analyses Do Not Justify Acquiring SPS

DOD has developed three economic analyses for SPS—one in 1995 and two updates (one in 1997 and another in 2000). While the initial analysis reflected a positive net present value, the two updates did not. Specifically, the 1997 analysis estimated life-cycle costs and benefits to be $2.9 billion and $1.8 billion, respectively, which is a recovery of only 62 percent of costs; the 2000 analysis showed even greater costs ($3.7 billion) and fewer benefits ($1.4 billion), which is a recovery of only 37 percent of costs (see fig. 2).

Nevertheless, these data were not reflected in the return-on-investment calculation in the analyses that were used as the basis for approving SPS. Instead, this return-on-investment calculation (1) included only those costs estimated to be incurred by the program office and (2) excluded the SPS implementation and operation and maintenance costs of DOD agencies and military services. According to program officials, the latter costs were excluded because either they would have been incurred anyway or the program office did not require them. For example, the officials stated that the DOD agencies and military services routinely upgrade their IT infrastructures to support existing systems; therefore, they assumed that the agencies and services would have purchased new infrastructures even if SPS had not been acquired. Also, program officials did not believe that training paid for by DOD agencies and military services should be included as a cost element because this is an elective expense (i.e., the program management office does not require this additional training). However, some DOD component officials told us that some of their infrastructure and other costs were being incurred solely to support implementation of SPS. Using DOD’s estimates, we calculated SPS’ net present value for fiscal years 1997 and 2000 to be about negative $174 million and negative $655 million, respectively.

DOD’s Office of Program Analysis and Evaluation is responsible for, among other things, verifying and validating the reliability of economic analyses for major programs, such as SPS, and providing its results to the program approval authority, which in this case is the DOD CIO. According to Office of Program Analysis and Evaluation officials, although the economic analyses were reviewed, there are no written results of these reviews. These officials stated, however, that they orally communicated concerns about the analyses to program officials and to DOD CIO officials responsible for program oversight and control. They also stated that while they could not recall specific issues discussed, they concluded that the economic analyses provided a reasonable basis for decisionmaking.

Latest SPS Economic Analysis Does Not Include Reliable Cost Data

To be useful for informed investment decisionmaking, analyses of project costs, benefits, and risks must be based on reliable estimates. However, most of the cost estimates in the latest economic analysis are estimates carried forward from the 1997 economic analysis (adjusted for inflation). Only the costs being funded and managed by the SPS program office, which are 13 percent of the total life-cycle cost in the analysis, were updated in 2000 to reflect more current contract estimates and actual expenditures/obligations for fiscal years 1995 through 1999. The costs to be funded and incurred by DOD agencies and the military services were not updated to account for all program changes or to incorporate better information. In its review of the 2000 economic analysis, the Naval Center for Cost Analysis also noted that the DOD agencies and the military services’ cost information, which accounted for the majority of the program’s overall costs, had not been updated.

In fact, only two cost elements were updated for the DOD component organizations in the 2000 economic analysis, and the estimates for these cost elements were based on estimates derived for just one service (the Air Force), and then extrapolated to all other DOD components. According to Departments of the Army, Navy, and Air Force component representatives, these original estimates of costs, as well as benefits, were highly questionable at best. However, this uncertainty was not reflected in the economic analysis by any type of sensitivity analysis (i.e., an analysis to explicitly present the return-on-investment implications associated with using estimates whose inherent imprecision could produce a range of outcomes). Such sensitivity analysis would disclose for decisionmakers the investment risk being assumed by relying on the calculations presented in the economic analysis.

According to the SPS program manager, costs in the 2000 economic analysis were not updated because information for the DOD components was not readily available for inclusion. Additionally, updating DOD component costs was not viewed as relevant because the return-on- investment calculation cited in the latest economic analysis did not include these costs, and the updated analysis was done after DOD leadership had decided to increase funding and continue the program. However, by not using economic analyses that are based on reliable cost estimates, DOD is making uninformed, and thus potentially unwise, multimillion-dollar investment decisions.

Latest SPS Economic Analysis Does Not Reflect Cost Increases and Benefit Delays

According to OMB guidance, analyses of investment costs, benefits, and risks should be (1) updated throughout a project’s life cycle to reflect material changes in project scopes and estimates and (2) used as a basis for ongoing investment selection and control decisions. To do less, risks continued investment in projects on the basis of outdated and invalid economic justification.

The latest economic analysis (January 2000) is outdated because it does not reflect SPS’ current status and known risks associated with program changes. For instance, this analysis is based on a program scope and associated costs and benefits that anticipated four software releases, each providing more advanced features and functions. However, according to the program manager, SPS now consists of seven releases over which additional requirements are to be delivered. Estimates of the full costs, benefits, and risks relating to these additional three releases are not part of this latest economic analysis.

Also, the 2000 economic analysis does not fully recognize actual and expected delays in meeting SPS’ full operational capability milestone. That is, the 2000 economic analysis assumed that this milestone would be September 30, 2003. However, as previously mentioned, this milestone date is unlikely to be met for a variety of reasons, such as user dissatisfaction with current system capabilities.

According to the SPS program manager, the latest economic analysis has not been updated to reflect changes because the analysis is not used for managing the program and because there is no DOD requirement for updating an economic analysis when changes to the program occur. By not ensuring that the program is being proactively managed on the basis of current information about costs, benefits, and risks, DOD is unnecessarily assuming an excessive amount of investment risk.

DOD Is Not Investing Incrementally in SPS

As we have previously reported, incremental investment management involves three fundamental components: (1) developing/acquiring a large system in a series of smaller projects or system increments, (2) individually justifying investment in each separate increment on the basis of costs, benefits, and risks, and (3) monitoring actual benefits achieved and costs incurred on completed increments and modifying subsequent increments or investments to reflect lessons learned.

While DOD is acquiring and implementing SPS in a series of incremental releases (originally four and now seven), it is not making decisions about whether to invest in each release on the basis of the release’s costs, benefits, and risks, and it is not measuring whether it is meeting cost-and- benefit expectations for each release that is implemented. Instead, DOD is treating investment in SPS as one, monolithic investment decision, justified by a single, all-or-nothing economic analysis. Moreover, DOD has not measured whether the incremental software releases have produced expected business value, even though its economic analysis aligns expected benefits with the then four incremental releases.

In June 2000, the SPS program office initiated a study in an attempt to validate the extent to which benefits would be realized as a result of DOD’s implementation of version 4.1 of the software. However, our review of the methodology and preliminary results revealed that the study was poorly planned and executed and that, while useful information may be compiled, DOD would be unable to use the study’s results to validate the accrual of benefits.

As a result, DOD will have spent hundreds of millions of dollars on the entire system before knowing whether it is producing value commensurate with cost. The program manager told us that knowing whether SPS is producing such value is not the program office’s objective. Rather, its objective is to simply acquire and deploy the system. Similarly, DOD CIO officials told us that although the economic analysis promised a business value that would exceed costs, DOD is not validating that implemented releases are producing that value because there is no DOD requirement and no metrics defined for doing so. By not investing incrementally in SPS, DOD runs the serious risk of discovering too late (i.e., after it has invested hundreds of millions of dollars) that SPS is not cost-beneficial.

Conclusions

DOD’s management of SPS is a lesson in how not to justify, make, and monitor the implementation of IT investment decisions. Specifically, DOD has not (1) ensured that accountability and responsibility for measuring progress against commitments are clearly understood, performed, and reported, (2) demonstrated, on the basis of reliable data and credible analysis, that the proposed system solution will produce economic benefits commensurate with costs before investing in it, (3) used data on progress against project cost, schedule, and performance commitments throughout a project’s life cycle to make investment decisions, and (4) divided this large project into a series of incremental investment decisions to spread the risks over smaller, more manageable components.

Currently, DOD is not effectively performing any of these basic tenets of effective investment management on SPS, and, as a result, DOD lacks the basic information needed to make informed decisions about how to proceed with the project.

Nevertheless, DOD continues to push forward in acquiring and deploying additional versions of SPS. Continuing with this approach to investment management introduces considerable risk. As a result, beyond possibly operating and maintaining already implemented releases for the remainder of fiscal year 2001 and meeting already executed contractual commitments, further investment in SPS has not been justified.

Recommendations for Executive Action

We recommend that the Secretary of Defense direct the Assistant Secretary of Defense for Command, Control, Communications, and Intelligence, as the designated approval authority for SPS, to clarify organizational accountability and responsibility for measuring SPS progress against commitments and to ensure that these responsibilities are met. We further recommend that the Secretary direct the Assistant Secretary to make investment in each new release, or each enhancement to an existing release, conditional upon (1) validating that already implemented releases of the system are producing benefits that exceed costs and (2) demonstrating on the basis of credible analysis and data that (a) proposed new releases or enhancements to existing releases will produce benefits that exceed costs and (b) operation and maintenance of already deployed releases of SPS will produce benefits that exceed costs.

Also, we recommend that the Secretary direct the Director, Program Analysis and Evaluation, to validate any analysis produced to justify further investment in SPS and to report any validation results to the Assistant Secretary of Defense for C3I. We also recommend that no further decisions to invest in SPS be made without these validation results.

Additionally, we recommend that the Secretary direct the Assistant Secretary of Defense for C3I to take the necessary actions, in collaboration with the SPS program manager, to immediately determine the current state of progress against program commitments addressed in this report and to ensure that such information is used in all future investment decisions concerning SPS.

Last, we recommend that the Secretary direct the Assistant Secretary of Defense for C3I to report by October 31, 2001, to the Secretary and to DOD’s relevant congressional committees on lessons learned from the SPS investment management experience, including what actions will be taken to prevent a recurrence of this experience on other system acquisition programs.

Agency Comments and Our Evaluation

In written comments on a draft of this report (reprinted in appendix II), the Acting Deputy Assistant Secretary of Defense for Command, Control, Communications, and Intelligence, who is also the DOD Deputy Chief Information Officer (CIO), agreed and partially agreed with our recommendations. In particular, the Deputy CIO agreed with our recommendation regarding the need to clarify organizational accountability and responsibility for measuring the program’s progress and ensuring that these responsibilities are met. The Deputy CIO also agreed to document lessons learned and have the Director of Program Analysis and Evaluation validate the results of any ongoing and future analyses of SPS’ return on investment.

However, the Deputy CIO disagreed with our report’s overall finding that continued investment in SPS has not been justified, and disagreed with those elements of our recommendations that could delay development and deployment of SPS, specifically, acquiring and using the information we believe is needed to make informed investment decisions. To support its position, however, the Deputy CIO offered no new facts or analyses. Instead, the comments either cite information already in our report or claims that the demands of incremental investment management are “inefficient, costly, and overly intrusive” and will cause “unwarranted delays and disruption to the program” for no other reason than “to satisfy economists and accountants.” According to DOD’s comments, the latest SPS economic analysis and the existing efforts to measure progress against selected program commitments provide sufficient bases for continuing to invest hundreds of millions of dollars in SPS. In particular, DOD stated that it is making progress in improving its ability to standardize contracting for goods and services, adding that this standardization progress is not only saving operating costs by retiring legacy procurement systems, but is also providing a standard environment within DOD for the exchange of information and a consistent look and feel of contract information to companies doing business with the department. In light of these outcomes, DOD commented that one of its main goals under the program is the timely fielding of SPS capability.

We disagree with these comments. As we describe in the report, incremental investment management practices are not only a best practice, but are also required by the Clinger-Cohen Act of 1996 and specified in OMB guidance and recently revised DOD acquisition policy. Therefore, DOD’s comments regarding incremental investment in SPS are at odds with contemporary practices and operative federal requirements and guidance.

Additionally, the economic analysis that DOD’s comments refer to is not reliable for a number of reasons that are discussed in our report. Specifically, this analysis treats SPS as a single, monolithic system investment. Experience has shown that such an all-or-nothing economic justification is too imprecise to use in making informed decisions on large investments that span many years. This kind of approach for justifying investment decisions has historically resulted in agencies’ investing huge sums of money in systems that do not provide commensurate benefits, and thus has been abandoned by successful organizations. Further, the need to avoid this pitfall was a major impetus for the Clinger-Cohen Act investment management reforms.

Also, as discussed in our report, the analysis highlights a return-on- investment calculation in its summary that does not include all relevant costs, such as the costs to be incurred by DOD components. Instead, the summary uses only SPS program office costs in this return-on-investment calculation. Further, this return-on-investment calculation does not reflect known changes in the program’s scope and schedule that would increase costs and reduce benefits. As our report points out, it does not, for example, reflect SPS’ change from four software releases to seven releases nor does it reflect the improbability of meeting a September 30, 2003, full operational capability date.

DOD’s comments also promote continued spending on SPS without sufficient awareness of progress against meaningful commitments, such as reliable data measuring and validating whether return-on-investment projections are being met. In fact, DOD’s comments emphasize standardization and fast deployment as core commitments. However, neither factor is an end in and of itself. Unless SPS provides the capability to perform procurement and contracting functions better and/or cheaper, and does so to a degree that makes SPS a more attractive investment relative to the department’s other investment options, DOD will not have adequate justification for investing further in SPS. As our report demonstrates, the department presently does not have the information it needs to know whether this investment is justified, and the information that is available raises serious questions about SPS’ acceptance by its user community and its business value. Nevertheless, DOD’s comments indicate its intention to implement SPS as planned. Our recommendations are aimed at ensuring that the department obtains the information it needs to make informed SPS investment decisions before proceeding with additional acquisitions.

DOD provided other clarifying comments that have been incorporated as appropriate throughout this report. The written comments, along with our responses, are reproduced in appendix II.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the date of this letter. At that time, we will send copies to the Chairmen and Ranking Minority Members of the Senate Committee on Armed Services; Senate Appropriations Subcommittee on Defense; House Armed Services Committee; House Appropriations Subcommittee on Defense; Subcommittee on Government Efficiency, Financial Management, and Intergovernmental Relations, House Committee on Government Reform; and Subcommittee on National Security, Veterans Affairs, and International Relations, House Committee on Government Reform. We are also sending copies of this report to the Director, Office of Management and Budget; the Secretary of Defense; the Acting Secretary of the Army; the Acting Secretary of the Navy; the Acting Secretary of the Air Force; the Acting Assistant Secretary of Defense Command, Control, Communications, and Intelligence/Chief Information Officer; the Under Secretary of Defense for Acquisition, Technology and Logistics; the Principal Deputy and Deputy Under Secretary for Management Reform; the Acting Director of Operational Testing and Evaluation; the Director of Program Analysis and Evaluation; the Director of Defense Procurement; the Director of the Defense Contract Management Agency; and the Director of the Defense Logistics Agency.

If you have any questions on matters discussed in this report, please call me at (202) 512-3439 or Cynthia Jackson, Assistant Director, at (202) 512-5086. We can also be reached by e-mail at hiter@gao.gov and jacksonc@gao.gov, respectively. Key contributors to this assignment are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine the progress that the Department of Defense (DOD) has made against the Standard Procurement System (SPS) program commitments and whether DOD has economically justified further investment in SPS.

To determine the progress made, we first analyzed relevant legislative and Office of Management and Budget (OMB) requirements, associated federal guidance, and applicable DOD policy and guidance on investment management. We then analyzed relevant program management documents and interviewed program officials to identify estimates and expectations for SPS’ cost, schedule, and performance, including the system capabilities to be provided and benefits to be produced by these capabilities. Source documents for this information included, but were not limited to, the acquisition strategy and program baseline, acquisition decision memorandums, and the quarterly Defense Acquisition Executive Summary report.

We then reviewed program management reports and briefings, interviewed program officials, and solicited information from the various DOD component organizations participating in SPS’ implementation to determine reported cost, schedule, and performance status. We compared this information against estimates and expectations to identify any variances. We did not independently validate the status information that we obtained. In cases where variances were found or status information was not available, we questioned program management and DOD’s Office of the Chief Information Officer (CIO) oversight officials. The DOD organizations that were part of our scope of contacts included the SPS program office within the Defense Contract Management Agency; the Office of the Director of Investments and Acquisition within the Office of the Assistant Secretary of Defense for Command, Control, Communications, and Intelligence (C3I)/Chief Information Officer; the Office of the Deputy Director (Strategic and Space Programs) within the Office of Program Analysis and Evaluation under the Office of the Undersecretary of Defense (Comptroller/Chief Financial Officer); the Office of Strategic and C3I Systems within the Office of the Director of Operational Test and Evaluation; and various offices within the Defense agencies and military services responsible for implementing SPS.

To determine whether DOD had economically justified SPS, we reviewed relevant legislative requirements and associated OMB guidance, as well as DOD policy and guidance on preparing and using economic analyses (cost, benefit, and risk), to measure progress against information technology (IT) investment decisions and to do so using an incremental or modular approach. We then obtained the original economic analyses prepared for the program and the two subsequent updates and evaluated them in light of relevant requirements, policies, and guidance to identify strengths and weaknesses. We also reviewed program management documents and interviewed program and oversight officials to understand how these analyses were reviewed and used, and we compared the results to relevant requirements and guidance. We also calculated the program’s net present value using the 1997 and 2000 economic analyses. In addition, we interviewed officials from the SPS program office, DOD CIO’s office, and DOD’s Program Analysis and Evaluation Office to discuss our results and seek clarifying information. We reviewed the methodology and preliminary results for the productivity study being conducted by DOD to substantiate the benefits to be realized by implementing SPS. We also interviewed officials from the SPS program office, Vector Research Incorporated, and Logistics Management Institute to discuss the methodology (e.g., survey execution, sampling, and analysis plans) and our conclusions on the study.

We conducted our work at DOD headquarters offices in Washington, D.C., and Alexandria, Virginia, and at American Management Systems, Incorporated, headquarters in Fairfax, Virginia, from October 2000 through June 2001 in accordance with generally accepted government auditing standards.

Appendix II: Comments From the Department of Defense

GAO Comments

1. See comments 2 through 9. 2. We did not independently validate DOD-reported data on the number of sites and procurement personnel who have received SPS training, the number of personnel who are located at sites where some version of SPS has been deployed, or the number and dollar value of contract actions completed in fiscal year 2000 using SPS; thus we have no basis to comment on the accuracy of these data. However, we do not agree with this comment’s thrust that these data points, combined with statements about DOD’s “improving its ability to standardize,” “providing a standard environment,” and providing “a consistent look and feel,” are sufficient measures of progress against commitments.

As the Clinger-Cohen Act and OMB guidance emphasize, and as we state in our report, investments in information technology need to contribute tangible, observable improvements in mission performance. Thus, standardization should not be viewed as an end in and of itself, but rather the means to an end, such as increased productivity and reduced costs. DOD’s comment on progress does not address such tangible, observable benefits. Instead, DOD states that SPS is saving operating costs by retiring legacy procurement systems, which, when SPS was initiated and justified, were to total 76 systems. However, as we also state in the report, only two legacy systems have been retired thus far as a result of the system’s being deployed to 773 sites, and DOD could not provide what, if any, savings were being realized by doing so. Moreover, the number of legacy systems that DOD eventually expects to be replaced by SPS has decreased to between 12 and 14. Further, while DOD states that SPS is providing standardization of contracting for goods and services for a segment of its procurement community, our report points out that each service is either in the process of developing, or has plans to develop, its own unique procurement policies, processes, and procedures. 3. For the reasons discussed in our report, we do not agree that DOD has justified further SPS investment in its 2000 economic analysis. For example, only the SPS costs funded and managed by the SPS program office, which are 13 percent of the total life-cycle cost in the analysis, were updated in 2000. The costs to be funded and incurred by DOD agencies and the military services were not updated to account for all program changes or to incorporate better information. Exacerbating this is the fact that only two cost elements were updated for the DOD component organizations in the 2000 economic analysis, and the estimates for these cost elements were based on estimates derived for just one service and extrapolated to all other DOD components. As another example, the analysis does not reflect the reduced number of legacy systems to be retired as well as recent evidence of user non- acceptance and non-use of the system, both of which drive benefit accrual.

We also do not agree that the analysis documented that a $163 million additional investment by the SPS program office would result in additional benefits of $389.5 million (in net present value terms). Rather, the analysis shows that acquiring, operating, and maintaining SPS over its life cycle will cost about $17 million more, but will produce about $390 million more in benefits (in net present value terms) than operating and maintaining legacy procurement systems. However, the analysis also shows that SPS as planned is not a cost- beneficial investment, because estimated costs exceed expected program benefits. 4. We do not disagree with DOD’s comments regarding the major program designation of SPS and the many organizations involved in the program. Also, while we agree that SPS program officials prepared the Acquisition Program Baseline and have reported quarterly against the commitments that are contained in this baseline, the baseline commitments and the associated reporting do not extend to all the relevant program goals and objectives that we cite in the report as needing to be measured in order to effectively manage a program like SPS, such as what the system is actually costing DOD and whether promised business value is actually being realized. Additionally, the Acquisition Program Baseline is dated May 4, 1998, and thus the commitments in this baseline are out of date.

We do not agree with DOD’s comments characterizing the timing of the 2000 economic analysis update. As we state in our report, this update was prepared after the increase in SPS funding had been approved. In fact, the Program Analysis and Evaluation official responsible for reviewing the analysis stated that it was for this reason that the review was perfunctory at best. 5. We do not agree with DOD’s comment that delaying investment in new SPS releases or enhancements until DOD validates that already implemented releases of the system are producing benefits in excess of costs is contrary to best practice and would delay and disrupt SPS in a way that is not warranted. As we state in our report, available evidence raises serious questions about the cost and benefit implications of users’ limited acceptance of already deployed versions as well as the cost implications of DOD’s limiting its maintenance options to a single vendor. Our point is that answers to these questions are needed in order to make informed investment decisions, and to proceed as planned with new investments without this information risks continuing to invest in the wrong system solution faster.

We agree with the comment that the program office initiated a productivity study in the summer of 2000. As we state in our report, this study was undertaken in response to DOD Inspector General findings that raised questions about user acceptance of the system. However, we do not agree that this study will substantiate the SPS benefit estimates and quantitatively document the benefits of SPS implementation through 2000 because the study’s scope and methodology are limited. For example: According to the program official responsible for the study, the purpose of the study is to estimate expected benefits to be realized in fiscal year 2003, from implementation of version 4.1.

The sample selected was not statistically valid, meaning that the results are not projectable to the population as a whole.

Relative to the other services, the Air Force was not proportionally represented in the study, meaning that any results would not necessarily be reflective of Air Force sites.

The study was based on the 1997 economic analysis instead of the more current 2000 economic analysis despite key differences between the two analyses. For example, the 1997 analysis shows 22 benefits valued at approximately $1.8 billion over the program’s 10-year life cycle, while the 2000 analysis contains only 19 benefits valued at approximately $1.4 billion.

According to SPS program officials, the survey instrument was not rigorously pre-tested. Such pre-testing is important because it ensures that the survey (1) actually communicates what it was intended to communicate, (2) is standardized and will be uniformly interpreted by the target population, and (3) will be free of design flaws that could lead to inaccurate answers.

The information being gathered does not map to the 22 benefit types listed in the 1997 SPS economic analysis. Instead, the study is collecting subjective judgments that are not based on predefined performance metrics for SPS capabilities and impacts. Thus, DOD is not measuring SPS against the benefits that it promised SPS would provide.

In addition, the senior official responsible for SPS implementation in the Air Force stated that the Air Force plans to conduct its own, separate survey to determine whether the system is delivering business value, indicating component uneasiness about the reliability of the SPS program office’s study. 6. We disagree. As we state in the report, incremental investment management practices are not only a best practice, but are also required by the Clinger-Cohen Act of 1996 and specified in OMB guidance and recently revised DOD acquisition policy. Therefore, DOD’s comments regarding incremental investment in SPS are at odds with contemporary practices and operative federal requirements and guidance.

Additionally, the economic analysis that DOD’s comments refer to is not reliable for a number of reasons that are discussed in our report. Specifically, this analysis treats SPS as a single, monolithic system investment. Experience has shown that such an all-or-nothing economic justification is too imprecise to use in making informed decisions on large investments that span many years. This kind of approach to justifying investment decisions has historically resulted in agencies investing huge sums of money in systems that do not provide commensurate benefits, and thus has been abandoned by successful organizations. Further, the need to avoid this pitfall was a major impetus for the Clinger-Cohen Act investment management reforms.

DOD’s comments also promote continued spending on SPS without sufficient awareness of progress against meaningful commitments, such as reliable data measuring and validating that return-on- investment projections are being met. In lieu of such measures, DOD’s comments emphasize standardization and fast deployment as core commitments. However, neither of these is an end in and of itself. Unless SPS provides DOD with the capability to perform procurement and contracting functions better and/or cheaper, and does so to a degree that makes SPS a more attractive investment relative to the department’s other investment options, DOD is not justified in investing further in SPS. As our report demonstrates, and as discussed in comments 2 and 3 above, DOD presently does not have the kind of reliable information it needs to know whether this investment is justified, and the information that is available raises serious questions about SPS’ acceptance by its user community and its business value.

With regard to the timely fielding of SPS, we note in our report that the program has already been delayed 3-1/2 years. In fact, delivery of version 4.1 of the software was 22 months overdue, and version 4.2 is already 5 months behind. While the impact of schedule delays and cost increases is a valid concern on any project, these factors are not the sole criteria. Introducing the wrong system solution faster and cheaper is still introducing the wrong solution no matter how it is presented. It is thus critically important that investment decisions be based on an integrated understanding of cost, benefit, and risk. 7. We do not dispute that the cited events have occurred, although we would add for additional context that we met with Assistant Secretary of Defense for Command, Control, Communications, and Intelligence (C3I) officials on March 15, 2001, the day before the memorandum requesting the first program review, to share our concerns and seek clarification, and that we provided our draft report to DOD for comment on May 25, 2001.

We do not agree with DOD’s comment that it is not necessary to have the Secretary of Defense direct the Assistant Secretary of Defense for C3I (DOD CIO) to determine the current state of SPS progress against commitments and to ensure that this information is used in future investiment decisions for several reasons.

First, the recent reviews cited in the DOD comments were for the Defense Contract Management Agency, which is the Component Acquisition Executive, and the Office of the Director of Defense Procurement, which is the SPS functional sponsor. Neither of these entities is the DOD CIO, who is the designated decision authority for SPS milestones and thus under SPS’ management structure has ultimate accountability for SPS.

Second, the recent reviews cited in DOD’s comments did not satisfy our recommendation for determining the current state of progress against the SPS commitments described in our report. In fact, we attended the April 27, 2001, review meeting, during which the senior attending official from the Defense Contract Management Agency stated that information being provided at this meeting was insufficient from a program management standpoint, lacking key information needed for informed SPS decision-making.

Third, the March 16, 2001, memorandum cited in DOD’s comments acknowledges the need to update SPS’ economic justification in light of the program’s cost and schedule changes and to ensure compliance with Clinger-Cohen Act requirements.

Fourth, the SPS program manager’s planned actions to respond to recent reviews are not sufficient to address the uncertainties surrounding SPS. According to the program manager, the acquisition program baseline would be updated to reflect the most recent program costs and expected schedule for full operational capability, but the program office had not planned any other actions.

Last, DOD’s comment stating that the Office of the DOD CIO and the Office of the Director of Defense Procurement plan to conduct an independent review of SPS within the next 180 days does not satisfy our recommendation because (1) DOD’s schedule for SPS calls for issuing contract task orders for subsequent SPS releases during this 6-month period and (2) this commitment is only a vague statement to “plan to conduct” a review at some undetermined, potentially distant, future point in time rather than having a review scheduled to occur in time to effect meaningful investment management improvements.

In light of DOD’s comments regarding this recommendation and for the reasons discussed above, we have modified our recommendation to specify that the recommended determination of the state of progress should occur immediately and should address each of the program commitments discussed in this report. 8. We acknowledge DOD’s agreement with the recommendation, but note that neither our recommendation nor DOD’s comment specifies when this report would be prepared. Accordingly, we have modified our recommendation to include a timeframe for reporting to the Secretary of Defense and relevant congressional committees on lessons learned and actions to prevent recurrence of those SPS experiences on other system acquisition programs.

Additionally, we disagree with DOD’s comments about the findings and conclusions in our report. In our view, the totality of evidence presented in our report, along with the results of prior Defense Inspector General reviews, supports our conclusion that SPS is a lesson in how not to justify, make, and measure implementation of investment decisions. Also, as addressed in comments 2 and 3, we do not agree with DOD’s point that SPS has been justified by the 1997 and 2000 economic analyses. Last, we do not agree with DOD’s comments that we incorrectly calculated a negative return on investment for SPS and that our methodology for calculating net present value is incorrect. To calculate net present value, we used current OMB guidance, which requires that relevant life-cycle cost estimates be used. Additionally, we used DOD’s own life-cycle cost estimates from its economic analyses. While we acknowledge that SPS officials told us that these life-cycle cost estimates included the costs of operating legacy procurement systems, we also requested that these officials identify what these legacy system costs are so that we could back them out. However, SPS officials told us that they did not know the amount of these costs. As a result, our calculation is based on the best information that the SPS program office had available and could provide. 9. See comments 3 and 8. Also, we agree that applying our net-present- value calculation methodology to the SPS and status quo cost-and- benefit data provided in the January 2000 economic analysis show that SPS is cheaper than the status quo option. However, this calculation also shows that SPS as planned is not cost beneficial. Also, DOD’s comments compare only a small portion of SPS life-cycle costs (program office investment costs) against the difference between expected benefits under the SPS scenario and the status quo scenario. This comparison is illogical because it assumes that an arbitrary part of relevant investment costs can be associated with the total benefit difference between alternatives. Accordingly, we do not agree with DOD’s comment.

While Appendix E of the January 2000 economic analysis contained some of the information provided in the tables contained in DOD’s comments, it did not provide a net present value calculation. Further, the Appendix E tables were not included in the economic analysis’ executive summary. Instead, the summary provided a benefits-to-costs ratio that excluded certain relevant costs.

Appendix III: GAO Contact and Staff Acknowledgment

GAO Contact

Acknowledgments

In addition to the person named above, Nabajyoti Barkakati, Harold J. Brumm, Jr., Sharon O. Byrd, James M. Fields, Sophia Harrison, James C. Houtz, Richard B. Hung, Barbarol J. James, and Catherine H. Schweitzer made key contributions to this report.

Related Reports

Standard Procurement System Use and User Satisfaction, Office of the Inspector General, Department of Defense (Report No. D-2001-075, March 13, 2001).

Defense Management: Actions Needed to Sustain Reform Initiatives and Achieve Greater Results (GAO/NSIAD-00-72, July 25, 2000).

Department of Defense: Implications of Financial Management Issues (GAO/T-AIMD/NSIAD-00-264, July 20, 2000).

Defense Management: Electronic Commerce Implementation Strategy Can Be Improved (GAO/NSIAD-00-108, July 18, 2000).

Initial Implementation of the Standard Procurement System, Office of the Inspector General, Department of Defense (Report No. 99-166, May 26, 1999).

Financial Management: Seven DOD Initiatives That Affect the Contract Payment Process (GAO/AIMD-98-40, January 30, 1998).

Allegations to the Defense Hotline Concerning the Standard Procurement System, Office of the Inspector General, Department of Defense (Report No. 96-219, September 5, 1996).