DOD's Standard Procurement System: Continued Investment Has Yet to Be Justified
Published: 2002-02-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) lacks management control of the Standard Procurement System (SPS). DOD has not (1) ensured that accountability and responsibility for measuring progress against commitments are clearly understood, performed, and reported; (2) demonstrated, on the basis of reliable data and credible analysis, that the proposed system solution will produce economic benefits commensurate with costs; (3) used data on progress against project cost, schedule, and performance commitments throughout a project's life cycle to make investment decisions; and (4) divided this large project into a series of incremental investment decisions to spread the risks over smaller, more manageable components.
section_title: What GAO Found
paragraphs: GAO found that DOD lacks the basic information needed to make informed decisions on how to proceed with the project. Nevertheless, DOD continues to push forward in acquiring and deploying additional versions of SPS. This testimony summarizes a July report (GAO-01-682).

section_title: Letter
paragraphs: I am pleased to be here today to discuss the Department of Defense’s  (DOD) management of its investment in the Standard Procurement System  or SPS program. The department launched this program a little more than  7 years ago with the laudable goal of replacing 76 existing procurement  systems with a single departmentwide system to more effectively support  divergent contracting processes and procedures across its component  organizations. Through SPS, the department expected to improve  efficiency and effectiveness in how it awarded and managed contracts,  and, at that time, estimated life-cycle costs to be approximately $3 billion  over a 10-year period.
The department’s goals for SPS are reinforced by the president’s recent  management agenda, which emphasizes investing in information  technology to achieve results. The agenda also noted that the federal  government has not produced measurable gains in productivity  commensurate with its investment in information technology, which is  now estimated to be more than $50 billion for fiscal year 2003. The agenda  reiterates that program performance and results are what matters most,  and that actual program accomplishments, as well as needs, should be the  prerequisite to continued funding. This emphasis is consistent with  information-technology investment management provisions of federal law  and guidance and information-technology management practices of  leading public- and private-sector companies.
For the SPS program, we reported in July 2001 that the department had  not met these investment management criteria. Specifically:
The department had not economically justified its investment in the  program because its latest (January 2000) analysis of costs and  benefits was not credible. Further, this flawed analysis showed that  the system, as defined, was not a cost-beneficial investment.
It had not effectively addressed the inherent risks associated with  investing in a program as large and lengthy as SPS because it had not  divided the program into incremental investment decisions that  coincided with incremental releases of system capabilities.
The department had not met key program commitments that were  used to justify the program. For example, the department committed  to implementing a commercially available contract management  system; however, because it had modified so much of the foundational  commercial product, SPS evolved into a customized DOD system.  Also, although the department committed to fully implementing the  system by March 31, 2000, this target date had slipped by 3 ½ years to  September 30, 2003, and program officials have recently stated that  this date will also not be met.
It did not know if it was meeting other key program commitments. For  example, the department had not measured whether promised system  benefits were being realized, and the information that was available  about system performance showed that users were not satisfied with  the system. Also, because DOD was not accumulating actual program  costs, it did not know the total amount spent on the program to date,  yet life-cycle cost projections had grown from about $3 billion to $3.7  billion.
Collectively, this meant that the question of whether further investment in  SPS was justified could not be answered with any certainty. Accordingly,  we recommended that investment in future releases or major  enhancements to the system be made conditional on the department first  demonstrating that the system was producing benefits that exceed costs,  and that future investment decisions be based on complete and reliable  economic justifications. We also recommended that program officials  clarify organizational accountability and responsibility for the program,  determine the program’s current status, and identify lessons learned from  the SPS investment management experience.
In commenting on a draft of our report, the Deputy Chief Information  Officer (CIO) generally disagreed with our recommendations, noting that  they would delay development and deployment of SPS. Since that time,  however, the department has either initiated or stated its intention to  initiate steps that are consistent with our recommendations. It has also  taken steps to address the findings of several department-sponsored  studies initiated at the time of our report. For example, it has (1) clarified  organizational accountability and responsibility for the program,  (2) established missing controls over key acquisition processes such as  requirements management and testing, and (3) begun addressing users’  concerns. In addition, department officials have stated that the department  will prepare an economic analysis before investing beyond already  executed contractual commitments and that it will conduct a productivity  study to assess the extent to which the department is deriving benefits  from SPS. These are positive steps that have advanced the program  beyond where it was at the time of our report.
Nevertheless, much remains to be done before the department will be in a  position to make an informed, data-driven decision about whether further  investment in the system is justified. Namely, although program officials  have stated their intentions to address our recommendations, they have  not yet committed to specific tasks for doing so nor have they established  milestone dates for completing these tasks. Further, the department may  expand the functionality of the current software release to include  requirements previously slated for later releases, which could compound  existing problems and increase costs; and, although intended to be a  standard system for the entire department, not all defense components  have agreed to adopt SPS.
subsections: section_title: SPS: A Brief Description and History
paragraphs: In November 1994, the Office of the Director of Defense Procurement  initiated the SPS program to acquire and deploy a single automated system  to perform all contract-management-related functions for all DOD  organizations. At that time, life-cycle costs were estimated to be about $3  billion over a 10-year period.
From 1994 to 1996, the department defined SPS requirements and solicited  commercially available vendor products for satisfying these requirements.  Subsequently, in April 1997, the department awarded a contract to  American Management Systems (AMS), Incorporated, to (1) use AMS’s  commercially available contract management system as the foundation for  SPS, (2) modify this commercial product as necessary to meet DOD  requirements, and (3) perform related services. The department also  directed the contractor to deliver functionality for the system in four  incremental releases. The department later increased the number of  releases across which this functionality would be delivered to seven,  reduced the size of the increments, and allowed certain more critical  functionality to be delivered sooner (see table 1 for proposed SPS  functionality by increment).
Since our report of July 2001, DOD has revised its plans. According to the  SPS program manager, current plans no longer include increments 6 and 7  or releases 5.0 and 5.1. Instead, release 4.2 (increment 5) will include at  least three, but not more than seven, subreleases. At this time, only the  first of the potentially seven 4.2 subreleases is under contract. This  subrelease is scheduled for delivery in April 2002, with deployment to the  Army and the Defense Logistics Agency scheduled for June 2002. Based on  the original delivery date, release 4.2 is about one year overdue.
The department reports that it has yet to define the requirements to be  included within the remaining 4.2 subreleases, and has not executed any  contract task orders for these subreleases. According to SPS officials, they  will decide later this year whether to invest in these additional releases.
As of December 2001, the department reported that it had deployed four  SPS releases to over 777 locations. The Director of Defense Procurement  (DDP) has responsibility for the SPS program, and the CIO is the  milestone decision authority for SPS because the program is classified as a  major Defense acquisition.
section_title: Numerous SPS Concerns Have Been Raised by Us and Others
paragraphs: Our July 2001 report detailed program problems and investment  management weaknesses. To address these weaknesses, we  recommended, among other things, that the department report on the  lessons to be learned from its SPS experience for the benefit of future  system acquisitions. Similarly, other reviews of the program  commissioned by the department in the wake of our review raised similar  concerns and identified other problems and management weaknesses. The  findings from our report are summarized below in two major categories:  lack of economic justification for the program and inability to meet  program commitments. We also summarize the findings of the other  studies.
subsections: section_title: DOD Had Not Economically Justified Its Investment in SPS
paragraphs: The Clinger-Cohen Act of 1996, OMB guidance, DOD policy, and practices  of leading organizations provide an effective framework for managing  information technology investments, not just when a program is initiated,  but continuously throughout the life of the program. Together, they  provide for  (1) economically justifying proposed projects on the basis of reliable  analyses of expected life-cycle costs, benefits, and risks; and  (2) using these analyses throughout a project’s life-cycle as the basis for  investment selection, control, and evaluation decisionmaking, and  doing so for large projects (to the maximum extent practical) by  dividing them into a series of smaller, incremental subprojects or  releases and individually justifying investment in each separate  increment on the basis of costs, benefits, and risks.
The department had not met these investment management tenets for SPS.  First, the latest economic analysis for the program—dated January 2000—  was not based on reliable estimates because most of the cost estimates in  the 2000 economic analysis were estimates carried forward from the April  1997 analysis (adjusted for inflation). Only the cost estimates being funded  and managed by the SPS program office, which were 13 percent of the  total estimated life-cycle cost in the analysis, were updated in 2000 to  reflect more current contract estimates and actual expenditures/  obligations for fiscal years 1995 through 1999. Moreover, the military  services, which share funding responsibility with the SPS program office  for implementing the program, questioned the reliability of these cost  estimates. However, this uncertainty was not reflected in the economic  analysis using any type of sensitivity analysis. A sensitivity analysis would  have disclosed for decisionmakers the investment risk being assumed by  relying on the estimates presented in the economic analysis.
Moreover, the latest economic analysis (January 2000) was outdated  because it did not reflect the program’s current status and known  problems and risks. For instance, this analysis was based on a program  scope and associated costs and benefits that anticipated four software  releases. However, as mentioned previously, the program now consists of  five releases, and subreleases within releases, in order to accommodate  changes in SPS requirements. Estimates of the full costs, benefits, and  risks relating to this additional release and its subreleases were not part of  the 2000 economic analysis. Also, this analysis did not fully recognize  actual and expected delays in meeting SPS’s full operational capability  milestone, which had been slipped by 3½ years and DOD officials say that  further delays are currently expected. Such delays not only increase the  system acquisition costs but also postpone, and thus reduce, accrual of  system benefits. Further, several DOD components are now questioning  whether they will even deploy the software, which would further reduce  SPS’s cost effectiveness calculations in the 2000 economic analysis.
Second, the department had not used these analyses as the basis for  deciding whether to continue to invest in the program. The latest  economic analysis showed that SPS was not a cost-beneficial investment  because the estimated benefits to be realized did not exceed estimated  program costs. In fact, the 2000 analysis showed estimated costs of $3.7  billion and estimated benefits of $1.4 billion, which was a recovery of only  37 percent of costs. According to the former SPS program manager, this  analysis was not used to manage the program and there was no DOD  requirement for updating an economic analysis when changes to the  program occurred.
Third, DOD had not made its investment decisions incrementally as  required by the Clinger-Cohen Act and OMB guidance. That is, although  the department is planning to acquire and implement SPS as a series of  five increments, it has not made decisions about whether to invest in each  release on the basis of the release’s expected return on investment, as well  as whether prior releases were actually achieving return-on-investment  expectations. In fact, for the four increments that have been deployed, the  department had not validated whether the increments were providing  promised benefits and was not accounting for the costs associated with  each increment so that it could even determine actual return on  investment.
Instead, the department had treated investment in this program as one,  monolithic investment decision, justified by a single, “all-or-nothing”  economic analysis. Our work has shown that it is difficult to estimate, with  any degree of accuracy, cost and schedule estimates for many increments  to be delivered over many years because later increments are not well  understood or defined. Also, these estimates are subject to change based  on actual program experiences and changing requirements. This “all-or- nothing” approach to investing in large system acquisitions, like SPS, has  repeatedly proven to be ineffective across the federal government,  resulting in huge sums being invested in systems that do not provide  commensurate benefits.
section_title: DOD Had Not Met or Did Not Know if It Had Met SPS Commitments
paragraphs: Measuring progress against program commitments is closely aligned with  economically justifying information-technology investments, and is equally  important to ensuring effective investment management. The Clinger- Cohen Act, OMB guidance, DOD policy,  and practices of leading  organizations provide for making and using such measurements as part of  informed investment decisionmaking.
DOD had not met key commitments and was uncertain whether it was  meeting other commitments because it was not measuring them. (See  table 2 for a summary of the department’s progress against commitments.) two analyses, such as the number and dollar value of estimated  benefits, and  the information gathered did not map to the 22 benefit types listed in  the 1997 economic analysis. Instead, the study collected subjective  judgments (perceptions) that were not based on predefined  performance metrics for SPS capabilities and impacts. Thus, the  department was not measuring SPS against its promised benefits.
The former program manager told us that knowing whether SPS was  producing value and meeting commitments was not the program office’s  objective because there was no departmental requirement to do so.  Rather, the objective was simply to acquire and deploy the system.  Similarly, CIO officials told us that the department was not validating  whether deployed releases of SPS were producing benefits because there  was no DOD requirement to do so and no metrics had been defined for  such validation. However, the Clinger-Cohen Act of 1996 and OMB  guidance emphasize the need to have investment management processes  and information to help ensure that information-technology projects are  being implemented at acceptable costs and within reasonable and  expected time frames and that they are contributing to tangible,  observable improvements in mission performance (i.e., that projects are  meeting the cost, schedule, and performance commitments upon which  their approval was justified). For programs such as SPS, DOD required this  cost, schedule, and performance information to be reported quarterly to  ensure that programs did not deviate significantly from expectations. In  effect, these requirements and guidance recognize that one cannot manage  what one cannot measure.
section_title: Other Studies Reported Similar Findings and Identified Other Concerns
paragraphs: Shortly after receiving our draft report for comment, the department  initiated several studies to determine the program’s current status, assess  program risks, and identify actions to improve the program. These  studies focused on such areas as program costs and benefits, planned  commitments, requirements management, program office structure, and  systems acceptance testing. Consistent with our findings and  recommendations, these studies identified the need to establish performance metrics that will enable the department to  measure the program’s performance and tie these metrics to benefits  and customer satisfaction; clearly define organizational accountability for the program; provide training for all new software releases; standardize the underlying business processes and rules that the  system is to support; acquire the software source code; and address open customer concerns to ensure user satisfaction.
In addition, the department found other program management concerns  not directly within the scope of our review, such as the need to appropriately staff the program management office with sufficient  resources and address the current lack of technical expertise in areas  such as contracting, software engineering, testing, and configuration  management; modify the existing contract to recognize that the system does not  employ a commercial-off-the-shelf software product, but rather is  based on customized software product; establish DOD-controlled requirements management and acceptance  testing processes and practices that are rigorous and disciplined; and assess the continued viability of the existing contractor.
section_title: DOD Has Begun Addressing Problems, But SPS’s Future Remains Uncertain
paragraphs: To address the many weaknesses in the SPS program, we made several  recommendations in our July 2001 report. Specifically, we recommended  that (1) investment in future releases or major enhancements to the  system be made conditional on the department first demonstrating that  the system is producing benefits that exceed costs; (2) future investment  decisions, including those regarding operations and maintenance, be  based on complete and reliable economic justifications; (3) any analysis  produced to justify further investment in the program be validated by the  Director, Program Analysis and Evaluation; (4) the Assistant Secretary of  Defense for Command, Control, Communications, and Intelligence (C3I)  clarify organizational accountability and responsibility for measuring SPS  program against commitments and to ensure that these responsibilities are  met; (5) program officials take the necessary actions to determine the  current state of progress against program commitments; and (6) the  Assistant Secretary of Defense for C3I report by October 31, 2001, to the  Secretary of Defense and to DOD’s relevant congressional committees on  lessons learned from the SPS investment management experience,  including what actions will be taken to prevent a recurrence of this  experience on other system acquisition programs.
DOD’s reaction to our report was mixed. In official comments on a draft of  our report, the Deputy CIO generally disagreed with our  recommendations, noting that they would delay development and  deployment of SPS. Since that time, however, the department has  acknowledged its SPS problems and begun taking steps to address some  of them. In particular, it has done the following.
The department has established and communicated to applicable DOD  organizations the program’s chain-of-command and defined each  participating organization’s responsibilities. For example, the Joint  Requirements Board was delegated the responsibility for working with  the program users to define and reach agreement on the needed  functionality for each software release.
The department has restructured the program office and assigned  additional staff, including individuals with expertise in the areas of  contracting, software engineering, configuration management, and  testing. However, according to the current program manager,  additional critical resources are needed, such as two computer  information technology specialists and three contracting experts.
It has renegotiated certain contract provisions to assume greater  responsibility and accountability for the requirements management  and testing activities. For example, DOD, rather than the contractor, is  now responsible for writing the test plans. However, additional  contract changes remain to be addressed, such as training, help-desk  structure, facilities support, and system operations and maintenance.
The department has designated a user-satisfaction manager for the  program and defined forums and approaches intended to better engage  users.
It has established a new testing process, whereby program officials  now develop the test plans and maintain control over all software  testing performed.
In addition, SPS officials have stated their intention to prepare analyses for future program activities beyond those already  under contract, such as the acquisition of additional system releases,  and use these analyses in deciding whether to continue to deploy SPS  or pursue another alternative; define system performance metrics and use these metrics to assess the  extent to which benefits have been realized from already deployed  system releases; and report on lessons learned from its SPS experience to the Secretary of  Defense and relevant congressional committees.
The department’s actions and intentions are positive steps and consistent  with our recommendations. However, much remains to be accomplished.  In particular, the department has yet to implement our recommendations  aimed at ensuring that (1) future releases or major enhancements to the  system be made conditional on first demonstrating that the system is  producing benefits that exceed costs and (2) future investment decisions,  including those regarding operations and maintenance, be based on a  complete and reliable economic justification.
We also remain concerned about the future of SPS for several additional  reasons. First, definitive plans for how and when to justify future system  releases or major enhancements to existing releases do not yet exist.  Second, SPS officials told us that release 4.2, which is currently under  contract, may be expanded to include functionality that was envisioned for  releases 5.0 and 5.1. Including such additional functionality could  compound existing problems and increase program costs. Third, not all  defense components have agreed to adopt SPS. For example, the Air Force  has not committed to deploying the software; the National Imagery and  Mapping Agency, the Defense Advanced Research Projects Agency, and  the Defense Intelligence Agency have not yet decided to use SPS; and the  DOD Education Agency has already adopted another system because it  deemed SPS too expensive.