Background

DLA is the Department of Defense’s (DOD) logistics manager for all DOD consumable items and some department repair items. Its primary business function is to provide supply support in order to sustain military operations and readiness. In addition to this primary function, which DLA refers to as either “materiel management” or “supply-chain management,” DLA performs five other major business functions: distributing materiel ordered from its inventory; purchasing fuels for DOD and the U.S. government; storing strategic materiel; marketing surplus DOD materiel for reuse and disposal; and providing numerous information services, such as item cataloging, for DOD, the United States, and selected foreign governments. DLA consists of a central command authority supported by a number of field commands that manage the agency’s six business functions.
Until about 1997, DLA generally developed its systems in-house. Since then, the agency has begun to acquire systems, relying on contractors for system development and managing the acquisition of these systems. Currently, DLA is in the process of acquiring two systems: Business Systems Modernization (BSM) and Fuels Automated System (FAS).
BSM is intended to modernize DLA’s materiel management business function, changing the agency from being solely a provider and manager of physical inventory to being a manager of supply chains. In this role, DLA would link customers with appropriate suppliers and track physical and financial business practices. It is planning to replace two large legacy systems, as well as several supporting programs, that are more than 30 years old and are not integrated. BSM is based on commercially available software products. DLA plans to acquire and deploy its BSM system solution through a series of four system releases/increments. First, it plans to demonstrate successful application of its new concept of doing business for selected commodities—namely, earth-moving equipment, medical/pharmaceutical supplies, and F/A-18 engine components—at the three Defense Supply Centers. If this first release is successfully demonstrated, DLA plans to expand the system solution to other commodities in three additional increments. DLA plans to invest approximately $658 million to acquire and implement BSM from fiscal years 2000 through 2005.
FAS is intended to help the Defense Energy Support Center manage about $5 billion in contracts with petroleum suppliers each year. FAS is to be a multifunctional system that provides for, among other things, point-of-sale data collection inventory control, finance and accounting, procurement, and facilities management. FAS, which relies on a commercially available software package, is being fielded incrementally. Increment 1 is the base- level operational module that is currently being deployed to base-level sites worldwide. The second increment is the enterprise-level system, which is to be deployed to its direct delivery commodity business unit. DLA plans to invest $293 million in FAS from fiscal year 1995 through 2002.
SEI’s SA-CMM is used to measure an organization’s capability to manage the acquisition of software. SEI’s expertise in, and model and methods for, determining software process assessment are recognized and accepted throughout the software industry. The model defines five levels of software acquisition maturity. Each level of maturity (except level 1) indicates process capability in relation to key process areas. For a maturity level to be achieved, all key process areas related to that level must be implemented effectively.
The second level of process maturity, level 2 (referred to as the repeatable level), demonstrates that basic management processes are established to track performance, cost, and schedule, and the necessary discipline is in place to repeat earlier successes on similar projects. Organizations that do not effectively implement all key process areas for the repeatable level are, by default, at level 1, the initial level of maturity. Level-1 processes can be described as immature, ad hoc, and sometimes chaotic; success in software acquisition for these organizations depends on the ability and commitment of the staff involved. Figure 1 further explains the five-level software acquisition model.
We evaluated DLA against six of the seven level-2 (repeatable) key process areas in the SA-CMM. We did not evaluate DLA on the seventh key process area—transition to support—because the contractors who are implementing BSM and FAS will support these systems when they are operational, rendering transition to support irrelevant for these acquisitions. We evaluated DLA against one level-3 (defined) key process area—acquisition risk management—because many software acquisition experts consider it to be one of the most important key process areas. These key process areas are described in table 1.
As established by the model, each key process area contains five common features—commitment to perform, ability to perform, activities to be performed, measurement and analysis of activities, and verification of activities’ implementation. These five features collectively provide a framework for the implementation and institutionalization of the key process areas. The common feature definitions are as follows:   Commitment to perform: This feature describes the actions that the organization takes to establish the process and ensure that it can endure. Key practices typically involve establishing organizational policies and sponsorship.
Ability to perform: This feature describes the preconditions that must exist in the project or organization to implement the software acquisition process competently. Key practices typically include assigning responsibility and providing training.
Activities to be performed: This feature describes the roles and procedures necessary to implement a key process area. Key practices typically involve establishing plans and procedures, performing the work, tracking it, and taking appropriate management actions.
Measurement and analysis of activities: This feature describes the steps necessary to measure progress and analyze the measurements. Key practices typically involve defining the measurements to be taken and the analyses to be conducted to determine the status and effectiveness of the activities performed.
Verification of activities’ implementation: This feature describes the steps the organization must take to ensure that project activities are performed in accordance with established processes. Key practices typically involve regular reviews by management.
Each common feature consists of a number of key practices—specific actions such as developing an organizational policy for software acquisition, developing various plans for software acquisition activities, and tracking a contractor’s progress. When an organization is evaluated against the SA-CMM, comparisons of actual performance against a key practice can result in one of four possible outcomes or ratings:   Strength: The key practice involved was effectively implemented.   Weakness: The key practice was not effectively implemented or was not implemented.
Observation: The key practice was evaluated, but cannot be characterized as a strength because (1) the project team did not provide sufficient evidence to support a strength rating or (2) the key practice was only partially performed.
Not rated: The key practice is not relevant to the project.
To achieve the repeatable level, DLA would have to demonstrate that the key practices related to this level were implemented effectively in the software acquisition projects being evaluated, and thus the project successes can be repeated in future projects.

	DLA Lacks the Capability to Acquire Software Effectively

DLA is not at level 2 (the repeatable level of maturity) when compared with the SA-CMM—meaning that DLA does not possess an agencywide or corporate ability to effectively acquire software-intensive systems. Whereas DLA’s BSM project fully or substantially satisfied SEI’s SA-CMM requirements for the key process areas for level 2, as well as requirements for one level 3 (defined level) key process area, its FAS project did not satisfy all the criteria for any of these key process areas. A discussion of how each system compared with the SA-CMM is summarized below.

		BSM Satisfied or Substantially Satisfied All Key Process Areas

BSM completely satisfied requirements for three of the level-2 key process areas, as well as for the one level-3 key process area, and substantially satisfied requirements for the remaining three level-2 key process areas that we evaluated. (See table 2 for the percentage of strengths and weakness for each area evaluated.) According to BSM officials, satisfying the criteria for the key process areas is attributable to the following factors: allocating adequate resources; following good program management practices, as defined in DOD Directive 5000; and working closely with relevant oversight groups. To address those few weaknesses that we identified, project officials told us that they have initiated corrective action.
BSM satisfied all key practices in   software acquisition planning, such as (1) having a written software acquisition policy, (2) having adequate resources for software acquisition planning activities, (3) developing and documenting the software acquisition strategy and plan, and (4) making and using measurements to determine the status of software acquisition planning activities.   project management, including (1) designating responsibility for project management, (2) having a written policy for the management of the software project, (3) having adequate resources for the duration of the software acquisition project, and (4) tracking the risks associated with cost, schedule, resources, and the technical aspects of the project.   contract tracking and oversight, including (1) designating responsibility for contract tracking and oversight, (2) including contract specialists in the project team, and (3) having a documented plan for contract tracking and oversight.   acquisition risk management, such as (1) having a risk management plan, (2) having a written policy for the management of software acquisition risk, and (3) measuring and reporting on the status of acquisition risk management activities to management.
BSM also satisfied all but one key practice in solicitation. Strengths included (1) designating responsibility for the software portion of the solicitation, (2) preparing cost and schedule estimates for the software products and services being acquired, and (3) having an independent review of cost and schedule estimates for the software products and services being acquired. BSM’s one weakness in this key process area was in not having a written policy for the software portion of the solicitation. This is significant because, according to the SEI, an institutional policy provides for establishing an enduring process.
BSM also satisfied all but three key practices in requirements development and management. Strengths included (1) having a written policy for managing the software-related contractual requirements, (2) having a group that is responsible for performing requirements development and management activities, and (3) measuring and reporting to management on the status of requirements development and management activities. One of the three weaknesses was the lack of a documented requirements development and management plan. Such a plan provides a roadmap for completing important requirements development and management activities. Without it, projects risk either not performing important tasks or not performing them effectively. The other two weaknesses involved the project office’s appraisal of system requirements changes. Specifically, BSM did not appraise (1) requests to change system requirements for their impact on the software being acquired or (2) all changes to the requirements for impact on performance and contract schedule and cost. These activities are critical to making informed, risk-based decisions about whether to approve requirements changes.
Last, BSM satisfied all but one key practice in evaluation, and we do not view that practice as significant. Strengths included (1) designating responsibility for contract tracking and oversight, (2) documenting evaluation plans and conducting evaluation activities in accordance with the plan, and (3) developing and managing evaluation requirements in conjunction with developing software technical requirements.
By generally satisfying these key process areas for its BSM project, DLA has increased the chances that the software acquired on this project will meet stated requirements and will be delivered on time and within budget.
See appendix II for more detailed information on key process areas and our findings on BSM.

		FAS Did Not Satisfy Any of the Key Process Areas

Because of the number and severity of its key practice weaknesses, FAS did not fully satisfy all the criteria for any of the five level-2 SA-CMM key process areas or for the one level-3 key process area that we evaluated.(See table 3 for the percentage of strengths and weakness for each area evaluated.) According to FAS officials, these weaknesses are attributable to a lack of adequate resources for the process areas. However, these officials stated that they are currently in the process of reorganizing and addressing resource shortages.
In the software-acquisition–planning key process area, FAS had 12 strengths, 2 weaknesses, and 1 observation. Strengths included, among other things, (1) having a written software acquisition policy, (2) developing and documenting the software acquisition strategy and plan, and (3) having management review software-acquisition–planning activities. Weaknesses included (1) not having adequate resources for software-acquisition–planning activities and (2) not measuring the status of the software-acquisition–planning activities and resultant products. The weaknesses are significant because they could prevent management from developing effective plans, from being aware of problems in meeting planned commitments, or from taking necessary corrective actions expeditiously.
In the requirements development and management key process area, FAS had six strengths, six weaknesses, and two observations. Examples of strengths included (1) having a written policy for managing the software- related contractual requirements and (2) having a group that is responsible for performing requirements development and management activities. However, we found weaknesses in important key practices that jeopardize effective control of the requirements baseline and can result in software products that do not meet cost, schedule, or performance objectives. Specific examples of weaknesses included (1) not having a documented requirements development and management plan, (2) not appraising requests to change system requirements for their impact on the software being acquired, (3) not appraising changes to the software-related contractual requirements for their impact on performance and contract schedule and cost, and (4) not measuring and reporting to management on the status of requirements development and management activities.
In the project management key process area, FAS had 10 strengths and 6 weaknesses. Strengths included, among other things, (1) designating responsibility for project management, (2) having a written policy for the management of the software project, and (3) using a corrective action system for identifying, recording, tracking, and correcting problems. Examples of weaknesses included (1) not having adequate resources for the duration of the software acquisition project, (2) not documenting the roles, responsibilities, and authority for the project functions, and (3) not tracking the risks associated with cost, schedule, and resources. These weaknesses are significant because they could jeopardize the project’s ability to ensure that important project management and contractor activities are defined, understood, and completed.
FAS had 11 strengths, 5 weaknesses, and 1 observation in the contract tracking and oversight key process area. Strengths included, among other things, (1) designating responsibility for contract tracking and oversight, (2) including contract specialists on the project team, and (3) ensuring that individuals performing contract tracking and oversight activities had experience or received training. Examples of weaknesses included (1) not having a documented plan for contract tracking and oversight and (2) not comparing the actual cost and schedule of the contractor’s software engineering effort with planned schedules and budgets. Because of these weaknesses, FAS contractor tracking and oversight activities are undisciplined and unstructured, thereby increasing the chances of FAS software acquisitions being late, costing more than expected, and not performing as intended.
In the evaluation key process area, FAS had nine strengths, two weaknesses, two observations, and two areas that were not rated. Strengths included, among other things, (1) designating responsibility for planning, managing, and performing evaluation activities, (2) documenting evaluation plans and conducting evaluation activities in accordance with the plan, and (3) developing and managing evaluation requirements in conjunction with developing software technical requirements. Weaknesses were (1) not ensuring that adequate resources were provided for evaluating activities and (2) not measuring and reporting on the status of evaluation activities to management. These weaknesses are significant because they preclude DLA decisionmakers from knowing whether contractor-developed software is meeting defined requirements.
FAS performed poorly in the one level-3 key process area that we evaluated—acquisition risk management—with 3 strengths, 11 weaknesses, and 1 observation. Examples of strengths included (1) having a written policy for the management of software acquisition risk and (2) designating responsibility for software acquisition risk activities. Weaknesses included, among others, (1) not having adequate resources for performing risk management activities, (2) not having a software risk management plan, and (3) not measuring and reporting on the status of acquisition risk management activities to management. Because of these weaknesses, the project office does not have adequate assurance that it will promptly identify risks and effectively mitigate them before they become problems.
By not satisfying any of these key process areas for its FAS project, DLA is unnecessarily increasing the risk that the software acquired on this project will not meet stated requirements and will not be delivered on time and within budget.
Appendix III provides more details on the key process areas and our findings on FAS.

	DLA Lacks Effective Software Process Improvement

The quality of the processes involved in developing, acquiring, and engineering software and systems has a significant effect on the quality of the resulting products. Accordingly, process improvement programs can increase product quality and decrease product costs. Public and private organizations have reported significant returns on investment through such process improvement programs. In particular, SEI has published reports of benefits realized through process improvement programs. For example, SEI reported in 1995 that a major defense contractor had implemented a process improvement program in 1988 and by 1995 had reduced its re-work costs from about 40 percent of project cost to about 10 percent, increased staff productivity by about 170 percent, and reduced defects by about 75 percent. According to a 1999 SEI report, a software development contractor reduced its average deviation from estimated schedule time from 112 percent to 5 percent between 1988 and 1996. During the same period, SEI reported that this contractor reduced its average deviation from estimated cost from 87 percent to minus 4 percent.
DLA does not currently have a software process improvement program, and recent efforts to establish one have not made much progress. We recently reported on DOD’s software process improvement efforts, including those within DLA. Specifically, we reported that before 1998, DLA had a software process improvement program; however, DLA eliminated it during a reorganization in 1998. In response to our report, DLA’s Chief Information Officer said that the software process improvement program was to be reestablished during fiscal year 2001 and that DLA’s goal would be for its system developers and acquirers to reach a level 2 on the CMM by fiscal year 2002.
To date, DLA has established an integrated product team for software process improvement that is tasked to study DLA’s software processes and, based on this study, to make recommendations on areas in which DLA needs to improve. DLA has also dropped its goal of achieving level 2 by 2002, and it does not intend to specify a CMM level for its contractors. The software process improvement team has produced several draft papers and a draft policy, but it does not have a plan or milestones for achieving software process improvement. According to an agency official associated with DLA’s process improvement effort, funding to develop and implement a software process improvement program has not been approved because of other agency IT funding priorities, such as BSM.

	Conclusions

DLA does not have the institutional management capabilities necessary for effectively acquiring quality software repeatedly on one project after another. This lack of agencywide consistency in software acquisition management controls means that software project success at DLA currently depends more on the individuals assigned to a given project than on the rules governing how any assigned individuals will function. That has proven to be a risky way to manage software-intensive acquisitions.
To DLA’s benefit, it currently has a model software acquisition project (BSM) that, albeit not perfect, is a solid example from which to leverage lessons learned and replicate effective software acquisition practices across the agency. To do so effectively, however, DLA will need to implement a formal software process improvement program and devote adequate resources to correct the weaknesses in the software acquisition processes discussed in this report. It will also have to commit the resources needed to implement a software process improvement program.

	Recommendations for Executive Action

To reduce the software acquisition risks associated with its two ongoing acquisition projects, we recommend that the Secretary of Defense direct the Director of DLA to immediately correct each BSM and FAS software- acquisition–practice weakness identified in this report.
To ensure that DLA has in place the necessary process controls to acquire quality software consistently on future acquisition projects, we recommend that the Secretary also direct the DLA Director to issue a policy requiring that (1) DLA software-intensive acquisition projects satisfy all applicable SEI SA-CMM level-2 key process areas and the level-3 risk management key process area and (2) DLA software contractors have comparable software process maturity levels; and   direct the Chief Information Officer (CIO) to establish and sustain a software process improvement program, including (1) developing and implementing a software process improvement plan that specifies measurable goals and milestones, (2) providing adequate resources to the program, and (3) reporting to the Director every 6 months on progress against plans.

	Agency Comments

DOD provided what it termed “official oral comments” from the Deputy Under Secretary for Logistics and Materiel Readiness on a draft of this report. In its comments, DOD stated that it generally concurred with the report and concurred with the recommendations. In particular, DOD stated that it will issue policy directives requiring the Director of DLA to (1) correct identified software acquisition practice weaknesses, except in circumstances in which corrections to past events make doing so impractical; (2) implement a plan in all software-intensive projects to satisfy all applicable SEI SA-CMM level-2 and level-3 key process areas, and require all DLA software contractors to have comparable software process maturity levels; and (3) establish and sustain a software process improvement program that includes a plan specifying measurable goals and milestones, provides adequate resources, and reports to the Director of DLA every 6 months on progress against the plan.
We are sending copies of this report to the Chairmen and Ranking Minority Members of the Senate Appropriations Subcommittee on Defense; the Subcommittee on Readiness and Management Support, Senate Committee on Armed Services; the House Appropriations Subcommittee on Defense; and the Subcommittee on Readiness, House Committee on Armed Services. We are also sending copies to the Director, Office of Management and Budget; the Under Secretary of Defense for Acquisition and Technology; the Deputy Under Secretary of Defense for Logistics and Materiel Readiness; and the Director, Defense Logistics Agency. Copies will be made available to others upon request.
If you have any questions regarding this report, please contact me at (202) 512-3439 or by e-mail at hiter@gao.gov. An additional GAO contact and staff acknowledgements are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine (1) whether the Defense Logistics Agency (DLA) has the effective software acquisition processes necessary to modernize and maintain systems and (2) what actions DLA has planned or in place to improve these processes.
To determine whether DLA has effective software acquisition processes, we applied the Software Engineering Institute’s (SEI) Software Acquisition Capability Maturity Model using our SEI-trained analysts. We focused on the key process areas necessary to obtain a repeatable level of maturity, the second level of SEI’s five-level model. We also evaluated against one level-3 key process area—acquisition risk management—because of its importance. We met with project managers and project team members to determine whether and to what extent they implemented each key practice, and to obtain relevant documentation. In accordance with the SEI model, for each key process area we reviewed, we evaluated DLA’s institutional policies and practices and compared project-specific guidance and practices against the required key practices.
More specifically, for each key practice we reviewed, we compared project-specific documentation and practices against the criteria in the software acquisition model. If the project met the criteria for the key practice reviewed, we rated it as a strength. If the project did not meet the criteria for the key practice reviewed, we rated it as a weakness. If the evidence was mixed or inconclusive and did not support a rating of either a strength or a weakness, we treated it as an observation. If the key practice was not relevant to the project, we did not rate it.
We evaluated DLA’s only two software acquisition projects underway at the time of our review: the Business Systems Modernization (BSM) and the Fuels Automated System (FAS).
To determine what actions DLA has planned or in place to improve its software processes, we identified the group within DLA that is tasked with performing this function. We interviewed agency officials who are involved in software process improvement, collected data, and analyzed draft policies and draft working papers describing planned work.
We performed our work from May through October 2001, in accordance with generally accepted government auditing standards.

Appendix II: Results of Software Acquisition Capability Maturity Model Evaluation for Business Systems Modernization

Appendix III: Results of Software Acquisition Capability Maturity Model Evaluation for Fuels Automated System

Appendix IV: GAO Contact and Staff Acknowledgments

	GAO Contact

	Acknowledgments

In addition to the individual named above, key contributors to this report were Suzanne Burns, Yvette Banks, Niti Bery, Sophia Harrison, Madhav Panwar, and Teresa Tucker.

Summary:
What GAO Found

The Defense Logistics Agency (DLA) plays a critical role in supporting America's military forces worldwide. DLA relies on software-intensive systems to support its work. An important determinant of the quality of software-intensive systems, and thus DLA's mission performance, is the quality of the processes used to acquire these systems. DLA lacks mature software acquisition processes across the agency, as seen in the wide disparity in the rigor and discipline of processes between the two systems GAO evaluated. DLA also lacks a software process improvement program to effectively strengthen its corporate software acquisition processes.