Information Technology: DOD's Acquisition Policies and Guidance Need to Incorporate Additional Best Practices and Controls
Released: 2004-07-30T00:00:00

Letter

Background

Best practices are tried and proven methods, processes, techniques, and activities that organizations define and use to minimize risks and maximize chances for success. As we have previously reported, using best practices can result in better outcomes—including cost savings; improved service and product quality; and, ultimately, a better return on investment. For example, two software engineering analyses of nearly 200 systems acquisitions projects indicated that teams using systems acquisition best practices produced cost savings of at least 11 percent over similar projects conducted by teams that did not employ the kind of rigor and discipline embedded in these practices. In addition, our research shows that best practices are a significant factor in successful acquisition outcomes, including increasing the likelihood that programs and projects will be executed within cost and schedule estimates.

DOD, GAO, and the Congress have all advocated the use of best practices. For example, in September 2000, DOD established a steering group to promote the use of systems acquisition best practices and lessons learned. Further, our 2001 report cited the benefits of DOD adoption of best practices and provided recommendations for establishing a mechanism for sharing best practices throughout DOD. In the fiscal year 2003 Defense Authorization Act, the Congress used our recommendations in directing DOD to expand its use of best practices. Specifically, it required the Under Secretary of Defense (Acquisition, Technology, and Logistics (AT&L)) and the Assistant Secretary of Defense (Command, Control, Communications, and Intelligence—now called Networks and Information Integration (NII))—to identify and serve as a clearinghouse for information regarding software acquisition best practices in the public and private sectors. In response, DOD assigned AT&L responsibility for serving as that clearinghouse. Further, the Defense Information Systems Agency created a Web site to provide information about acquisition best practices.

DOD Relies Extensively on IT Systems to Perform a Variety of Business Functions

DOD is one of the largest and most complex organizations in the world. In fiscal year 2003, DOD reported that its operations involved over $1 trillion in assets, nearly $1.6 trillion in liabilities, disbursements of more than $416 billion, and approximately 3.3 million military and civilian personnel. Execution of these operations spans a wide range of defense organizations, including the military services, defense agencies and field activities, and various combatant and joint operation commands. To execute these military operations, the department performs an assortment of business functions, including logistics management, procurement, healthcare management, and financial management.

To support its business functions, DOD reports that it currently relies on about 2,300 IT systems, including accounting, acquisition, logistics, and personnel systems. Moreover, its future investment in business systems is expected to be sizable. For fiscal year 2004, DOD requested approximately $28 billion in IT funding to support a wide range of military and business operations. Approximately $9 billion of this amount is to support primarily command and control systems, and the remaining $19 billion is to support the operation, maintenance, and modernization of business systems.

Overview of DOD’s Acquisition Management Framework

Since the 1980s, DOD’s oversight of its systems acquisitions had been defined by a series of three documents—commonly called the 5000 series—that provided the policies and guidance for departmental efforts to acquire service capabilities and systems: DOD Directive 5000.1, The Defense Acquisition System—describes the management principles for DOD’s acquisition programs.

DOD Instruction 5000.2, Operation of the Defense Acquisition System—outlines the framework for managing acquisition programs.

DOD 5000.2-R, Mandatory Procedures for Major Defense Acquisition Programs (MDAPS) and Major Automated Information System (MAIS) Acquisition Programs—provides the mandatory procedures for acquiring major defense programs.

These documents have been revised several times. Most recently, in October 2002, the Deputy Secretary of Defense determined that the existing versions of these three documents required further revisions to improve acquisition efficiency, flexibility, creativity, and innovation. As a result, the Deputy Secretary canceled the existing versions of each document and instructed the Under Secretary for AT&L; the Assistant Secretary for NII; and the Director, Operational Test and Evaluation (OT&E) to jointly revise the documents. (Table 1 describes selected responsibilities of these three entities.) The revised directive and instruction were issued in May 2003. Both were shortened and modified to focus on required outcomes and legal requirements and to eliminate the “how-to” details in the previous versions. In doing so, DOD intended the revisions to provide program managers with more flexibility in executing their respective programs.

DOD 5000.2-R was renamed the Interim Defense Acquisition Guidebook and was made optional guidance on best practices and lessons learned. (See fig. 1.) According to DOD officials, improvements to the guidebook are currently under way. Until they are completed, DOD 5000.2-R serves as the guidebook.

According to the revised policy, an acquisition is a directed, funded effort that provides a new, improved, or continuing materiel, weapon or information system, or service capability. The revised 5000 series applies to acquisitions conducted by all of the department’s organizational components. These components include the military services and the defense agencies, such as the Defense Information Systems Agency.

The 5000 series describes a management framework that is intended to translate mission needs and requirements into systems acquisition programs. To accomplish this, the framework specifies five phases: Concept refinement: Intended to refine the initial system concept and produce a strategy for acquiring a system capability. A decision is made at the end of this phase (milestone A decision) whether to move to technology development.

Technology development: Intended to determine the appropriate set of technologies to be integrated into the system by iteratively assessing the viability of various technologies while simultaneously refining user requirements. Once the technology has been demonstrated in a relevant environment, a decision is made at the end of this phase (milestone B decision) whether to move to system development and demonstration.

System development and demonstration: Intended to develop a system or a system increment and demonstrate through developer testing that the system/system increment can function in its target environment. A decision is made at the end of this phase (milestone C decision) whether to move to production and deployment.

Production and deployment: Intended to achieve an operational capability that satisfies the mission needs, as verified through independent operational test and evaluation, and ensures that the system is implemented at all applicable locations.

Operations and support: Intended to provide a support program to meet operational support requirements and sustain the system in the most cost-effective manner over its total life cycle.

According to the framework, an acquisition program may begin at milestone A, B, or C, and its progress depends on obtaining sufficient knowledge to make an informed decision about whether to continue to the next acquisition phase. Although the framework permits programs to be managed as a single project, DOD Instruction 5000.2 states that the department prefers an evolutionary acquisition strategy that delivers a mature product in increments. Under such a strategy, the instruction states that each increment is to begin with a milestone B decision, and the production and deployment phase of each increment is to begin with a milestone C decision. Figure 2 provides a simplified diagram of the department’s acquisition management framework.

Typically, the Under Secretary for AT&L or a designee serves as the investment decision authority for DOD acquisitions, but the Assistant Secretary for NII/Chief Information Officer or a designee serves as the decision authority for IT systems acquisitions.

Past Evaluations of DOD Business Systems Have Revealed Acquisition Management Weaknesses

Due in part to its long-standing and pervasive IT acquisition management weaknesses, DOD has had limited success in acquiring IT resources to replace its outdated business systems. Both inspector general and departmental studies have cited these weaknesses on a variety of acquisition projects. We have also reported on business systems acquisition weaknesses. For example, in 2002 we reported that the Defense Logistics Agency did not have effective corporate processes for consistently acquiring software (the most costly and complex component of systems), and that the agency did not have a software process improvement program in place to effectively strengthen its software acquisition processes. In 2002, we also reported acquisition management problems with the Military Health System’s acquisition of DOD’s primary medical information system, including weaknesses in incremental economic justification, risk management, and contract management. In 2003, we reported that DOD had not economically justified four finance and accounting systems that have an estimated cost of more than $1 billion. In each of our reports, we have made recommendations for strengthening acquisition management through the adoption of best practices. DOD has largely agreed with our recommendations, but its progress to date in implementing them across the department has been uneven.

DOD’s Acquisition Policy and Guidance Are Consistent with Some, but Not All, Key Acquisition Best Practices

We and others, such as Carnegie Mellon University’s Software Engineering Institute, have identified and promoted the use of a number of best practices associated with acquiring IT systems. For the purposes of this report, we have identified 18 relevant best practices and grouped them into two categories: (1) 10 best practices for acquiring any type of business system and (2) 8 complementary best practices that relate specifically to acquiring commercial component-based business systems. Examples of best practices relevant to any business systems acquisition include ensuring that (1) reasonable planning for all parts of the acquisition occur, (2) a clear understanding of system requirements exists, and (3) risks are proactively identified and systematically mitigated. Examples of best practices relevant to commercial component-based systems acquisitions include ensuring that (1) commercial product modification is effectively controlled, (2) relationships among commercial products are understood before acquisition decisions are made, and (3) the organizational impact of using new system functionality is proactively managed. Each of these practices is composed of from one to eight activities and is described in table 2. DOD officials responsible for revising the 5000 series told us that each of these 18 practices are relevant to DOD business systems acquisitions. Appendix II provides additional details on each of these practices.

DOD’s acquisition policies and guidance largely incorporate the 10 best practices that are relevant to any business systems acquisition. More specifically, they fully incorporate 7 of the 10 practices and partially incorporate the other 3 practices. However, they generally do not incorporate the 8 best practices that relate to acquiring commercial component-based business systems. In particular, they fully incorporate 1 best practice, partially incorporate 2, and do not incorporate the remaining 5. (See fig. 3 for a summary of our analysis.) At our request, DOD officials responsible for the 5000 series also assessed it against those 18 practices, and we incorporated information from their assessment into ours. These officials also told us that the acquisition guidebook is currently being expanded to incorporate additional best practices, but they did not provide us with a plan for accomplishing this. Until this is accomplished, DOD is increasing the risk that important and beneficial best practices will not be followed and that DOD business systems investments will not deliver promised capabilities and benefits on time and within budget.

The 5000 Series Largely Incorporates Best Practices Relevant to Any Business Systems Acquisition

Of the 10 best practices that we categorized as relevant to the acquisition of any business system, whether custom-developed or developed using commercial packages and products, essentially all have been incorporated into DOD’s acquisition policies and guidance. (See table 3 for our detailed comparative analysis of the 5000 series against the 10 best practices.) For example, those practices aimed at ensuring that the acquisition is well planned, that the system is adequately tested and evaluated against contractual requirements, and that the requirements are clearly defined and understood by all stakeholders are all addressed in the 5000 series. Similarly, for the 3 practices that are not fully addressed, this is the case because one activity associated with the practice is not addressed. According to DOD officials responsible for revising the 5000 series, the policies contain those best practices mandated by either law or DOD regulation, and other, optional best practices are contained in the interim guidance represented by the former DOD 5000.2-R.

Nevertheless, the activities that are missing from the 3 practices are important, and their absence increases the risk that the activities, and thus the practice, will not be adequately performed. In turn, this increases the risk that acquisition projects will fall short of expectations. The best practice aimed at ensuring that risks are proactively identified and systematically mitigated is a case in point. This practice has five activities associated with it, one of which the 5000 series does not address—project reviews include the status of identified risks. As with all the activities under this practice, this activity plays an important role in ensuring that the appropriate level of attention and visibility is regularly given to risk identification and mitigation to ensure that it is effectively executed. Conversely, if the activities are not provided for in policy and guidance, it is unlikely that they will be performed, and it is likely that acquisition risks will become cost, schedule, and performance problems.

The 5000 Series Generally Does Not Incorporate Best Practices Relevant to Commercial Component-Based Business Systems Acquisitions

Of the 8 best practices relevant to acquiring commercial component-based business systems, few have been incorporated into DOD systems acquisition policies and guidance. (See table 4 for our detailed comparative analysis of the 5000 series against the 8 best practices.) For example, while the practice aimed at ensuring that adequate planning takes place for integrating commercial products with legacy systems is incorporated into the 5000 series, practices associated with closely controlling any modification to the software of these packages and products, thoroughly analyzing and understanding the dependencies among commercial products before acquiring them, and proactively managing the institutional change that results from implementing the functionality in commercial packages and products are not incorporated. According to DOD officials responsible for revising the 5000 series, these practices were not included in the recently revised version of DOD’s acquisition policies because they included only those in existing law or regulation.

Nevertheless, the absence of these practices from the 5000 series increases the risk that the practices will not be performed, which, in turn, increases the risk that acquisition projects will fall short of expectations. The practice intended to ensure development of a quality solicitation and selection of a best-qualified contractor illustrates this. Specifically, this practice calls for contract bidders to be evaluated on their ability to implement commercial components. This evaluation is important because integrating and implementing these component products is sufficiently different from developing customized system solutions; it requires different core competencies and experiences to be successful. By explicitly taking this into consideration in evaluating and selecting a contractor, the risk of contract award to a less-than-best-qualified contractor is reduced.

DOD Officials Report That They Are in the Process of Revising the Interim Defense Acquisition Guidebook

The DOD officials responsible for revising the 5000 series told us they recognize the need for the 5000 series to incorporate additional best practices. To this end, they reported that efforts are under way to expand the Interim Defense Acquisition Guidebook to include additional best practices and lessons learned across the department. However, the officials could not provide us with a documented plan and associated documentation showing how this task will be accomplished, what resources are needed and assigned to accomplish it, when it will be accomplished, and where the department stands in accomplishing it. Instead, the officials told us that progress on it has been slowed by other priorities, such as the need to first revise DOD Directive 5000.1 and DOD Instruction 5000.2. They also said that there is only a small number of staff available to work on what they described as being an extensive revision of the guidebook. According to these officials, 80 to 90 percent of the revision has been completed and reviewed and their goal is to publish the initial version of the revised guidebook by September 30, 2004. In our view, until the missing best practices that we cite in this report are included in DOD’s acquisition policies and guidance, the chance that business systems acquisitions will follow the policies and guidance and consistently produce a successful outcome is diminished.

DOD’s Acquisition Policies Do Not Contain Sufficient Controls to Ensure That the Requirement Is Met for Appropriately Applying Best Practices

Federal laws and regulations define the need for effective controls over agency programs, and controls are a key factor in achieving program results, minimizing operational problems, and managing evolving demands and priorities. Controls over defined processes, procedures, and support activities are considered effective if they entail measuring and verifying whether a given practice is followed. Without sufficient controls, it is unlikely that practices will be consistently employed, which, in turn, increases the probability that the positive program and project outcomes these practices are designed to produce will not occur.

DOD’s acquisition policy requires program managers and investment decision authorities to examine and, as appropriate, adopt best practices. However, neither the policies nor the accompanying guidance explain what “examine” means, including whether practice use is to be measured and verified. Instead, the policies state that any issues regarding the intent of the 5000 series, which would include whether practice adoption is to be measured and validated, shall be resolved by the investment decision authority, meaning that it is entirely up to this individual what information relative to the use of best practices is relevant and necessary to ensure that best practices are appropriately followed. According to the Chairman of the Defense Acquisition Policy Working Group, this control is sufficient, and explicit requirements for measuring and validating the use of best practices are not necessary.

In our view, not requiring that decision authorities examine the measurement and validation of best practices’ use increases the chance that important best practices will not be appropriately followed, as required by DOD policy. As we have previously reported, a lack of explicit controls that require review of relevant information at key decision points raises the risk of making uninformed project decisions, as well as the risk that investments will not meet cost, schedule, capability, and benefit commitments.

Conclusions

DOD recognizes the importance of business systems acquisition best practices by including best practices in their revised policy and guidance. However, other practices that, if followed, could increase the odds of acquisitions delivering promised system capabilities and benefits on time and within budget have yet to be similarly included. In particular, those practices associated with the successful acquisition of commercial component-based business systems have not been sufficiently incorporated into either the policies or the guidance. Moreover, effective controls for ensuring that best practices are appropriately followed are not adequately provided for in the policies. Although DOD officials intend to expand the coverage of best practices in future versions of DOD’s acquisition guidance, it is unclear what the scope, nature, and status of these intentions are because explicit plans for revising the guidance and associated progress reports were not available. Until DOD incorporates the best practices we found missing in the 5000 series, and until it strengthens the means by which the appropriate use of these practices will be ensured, its business systems acquisitions will be exposed to unnecessary risk. Therefore, it is important for DOD to treat further revisions of its acquisition policy and guidance relative to business systems as a priority and move quickly to incorporate missing best practices and associated controls for ensuring that the practices are followed.

Recommendations for Executive Action

To improve DOD’s ability to acquire business systems, we recommend that the Secretary of Defense direct the Under Secretary for AT&L, in collaboration with the Assistant Secretary for NII and the Director, OT&E, to take the following actions: 1. Develop and implement an explicit plan for incorporating into the 5000 series the best practices and associated activities currently missing from the series. We recommend that the plan specify tasks to be performed, resources needed and assigned, and milestones for completing tasks. 2. We further recommend that progress against this plan be tracked and reported as appropriate, and that the plan, at a minimum, incorporate each of the following best practice activities: Product line requirements—rather than just the requirements for the system being acquired—are an explicit consideration in each acquisition.

Acquisition project management activities are communicated to all stakeholders.

Acquisition reviews include the status of identified risks.

Modification of commercial components is discouraged and allowed only if justified by a thorough analysis of life-cycle costs and benefits.

Modification or upgrades to deployed versions of system components are centrally controlled, and unilateral user release changes are precluded.

Acquisition decisions about commercial components are based on deliberate and thorough research, analysis, and evaluation of the components’ interdependencies.

Acquisition plans provide for preparing users for the impact that the business processes embedded in the commercial components will have on their respective roles and responsibilities.

Changes affecting how users will be expected to use the system to execute their jobs are actively managed.

Systems integration contractors are explicitly evaluated on their ability to implement commercial components.

Investment decisions throughout a system’s life cycle are based on a continuous set of tradeoffs among capabilities available in commercial components (current and future), the architectural environment in which the system is to operate, defined system requirements, and existing cost/schedule constraints.

Evaluation criteria are established for selecting among commercial component options that include both defined system requirements and vendor/commercial product characteristics. 3. To ensure that the best practices provided for in DOD acquisition policies and guidance are appropriately followed, we also recommend that the above recommended plan incorporate steps to include in DOD’s acquisition policies a provision for measurement and verification of best practices.

Agency Comments and Our Evaluation

In written comments on a draft of this report, signed by the Principal Director for Command, Control, Communications, Space, and Information Technology Programs in the office of the DOD Assistant Secretary for Networks and Information Integration, DOD agreed with the importance and relevance of the best practices that we cite in the report. Additionally, DOD agreed with 2 of our 13 recommendations for incorporating additional best practices, stating that the department would incorporate the 2 practices in its policies and guidance.

DOD also partially agreed with 9 of our recommendations for incorporating additional practices, stating that it would consider augmenting its coverage of 5 of the practices and that it believed that 4 practices already existed in its policies and guidance. With regard to the 5 practices, DOD stated that it needed to review each practice further and determine the need for its emphasis or endorsement in the 5000 series. We understand DOD’s desire to carefully consider changes to its acquisition policies and guidance, and believe that such careful deliberation is consistent with the spirit of our recommendations.

With regard to the remaining 4 practices that DOD partially agreed with, we do not agree with the department’s comment that these best practices adequately exist in the 5000 series. For example, DOD commented that because its existing policies and guidance provide for the use of integrated product teams, which, according to DOD, are a means for promoting collaboration and facilitating communication among stakeholders, its policies and guidance therefore already provide for communicating information about management of a given acquisition project to all relevant project stakeholders. While we do not question the use of integrated product teams as a way to communicate information, the point of our recommendation is that there needs to be an explicit recognition in policy or guidance of the type of information to be communicated and with whom it is to be communicated. Restated, our recommendation for incorporating the best practice of communicating acquisition management activities to all stakeholders is intended to permit communication vehicles, such as integrated product teams, to be more effective by explicitly providing for this best practice in relevant policies and guidance. As another example, we do not agree with DOD’s comment that 2 of the best practices that we recommended for incorporation in its policies and guidance—preparing users for the impact that business process changes embedded in commercial components will have on their roles and responsibilities, and actively managing changes in how users will use new systems—are already sufficiently contained in the 5000 series. In particular, while we agree that the series references an acquisition management toolkit that addresses these 2 best practices, this reference is provided only once in the 5000 series, and this reference is only in relation to one phase of the acquisition cycle (the technology development phase). Given the importance and relevance of these practices to successful implementation of commercial component-based systems, our position, and thus the basis for our recommendations, is that the practices’ implementation would be more likely to occur if the practices were visible and better recognized in all relevant stages of DOD’s acquisition cycle.

Also in its comments, DOD did not agree with our recommendations to develop and implement an explicit plan to govern its ongoing and future policy and guidance revision activities, specifically stating that the recommendation was inappropriate and offering updated information on the status of and associated milestone for completing its activities. While we have updated our report to include the revised status and milestone information, we do not agree with DOD that a plan governing these efforts is not needed. Given the importance of DOD's acquisition policies and guidance, and the need for their continuous review and update to reflect new acquisition best practices, we believe that having an explicit plan that defines how and when these policies and guidance will be incorporated is essential. Among other things, a plan would highlight the resource constraints that this revision effort has been subject to, would allow measurement against defined milestones, and would allow disclosure of progress and impediments.

DOD also did not agree with our recommendation to add stronger controls for ensuring adherence to the best practices that are contained in its acquisition policies and guidance, stating that its existing oversight process includes the necessary compliance activities. We disagree. As we state in the report, DOD’s existing policy leaves these compliance activities to the discretion of the program manager and the investment decision authority, and it does not provide for measurement and verification of the use of best practices, both of which are recognized components of effective control processes.

A copy of DOD’s comments is reprinted in appendix III, along with our response.

We are sending copies of this report to the Chairmen and Ranking Minority Members of the Senate and House Committees on Armed Services; Subcommittees on Defense, Senate and House Committees on Appropriations; and the Subcommittee on Military Readiness, House Committee on Armed Services. We are also sending copies to the Director, Office of Management and Budget; the Secretary of Defense; the Under Secretary of Defense (AT&L); the Assistant Secretary of Defense (NII)/Chief Information Officer; and the Director, OT&E. We will make copies available to others on request. This report will also be available at no charge on our Web site at http://www.gao.gov.

If you or your staff has any questions concerning this report, please contact me at (202) 512-3439. I can also be reached by e-mail at hiter@gao.gov. Other contacts and key contributors to this report are listed in appendix IV.

Objectives, Scope, and Methodology

Our objectives were to determine whether the Department of Defense’s (DOD) revised systems acquisition policies for acquiring information technology (IT) business systems (1) are consistent with industry best practices, including those pertaining to commercial component-based systems, and (2) provide the necessary controls to ensure that the department’s component organizations adhere to the practices.

To accomplish the first objective, we identified the DOD policies and guidance relevant to business systems. These policies and guidance are contained in three documents—DOD Directive 5000.1, DOD Instruction 5000.2, and the Interim Defense Acquisition Guidebook—and are generally referred to as the 5000 series. We then reviewed each of these documents and discussed with DOD officials responsible for developing and revising the documents what steps were taken to incorporate best practices into each document. The DOD officials that we interviewed were from the offices of the Under Secretary of Defense (Acquisition, Technology, and Logistics (AT&L)); the Assistant Secretary of Defense (Networks and Information Integration (NII)); and the Director, Operational Test and Evaluation (OT&E). Next, we researched prior GAO reports; the work of federally funded research and development organizations, such as the Software Engineering Institute and The Aerospace Corporation; and other authoritative sources to identify business systems acquisition best practices. Our research produced 18 best practices, including associated activities, that we placed into two categories—one category for the practices that are relevant to any business systems acquisition and one category for the practices that are relevant to commercial component-based business systems acquisitions. In particular, we drew extensively from the Software Engineering Institute’s Software Acquisition Capability Maturity Model. In doing so, we selected practices from the model’s repeatable level of process maturity, which is level two on the model’s five-level scale. We used the repeatable level of process maturity because it is intended to provide the necessary process discipline to allow an organization to repeat earlier successes on similar projects. In addition, we included one Software Acquisition Capability Maturity Model level-three process area—risk management—because many experts consider it to be one of the most important process areas. We did not attempt to develop an exhaustive list of best practices and, in fact, fully recognize that additional best practices exist, such as ensuring that the appropriate level of human capital knowledge, skills, and abilities are employed, as well as that additional activities for the practices that we have identified exist, such as those configuration management activities associated with identifying, controlling, reporting on, and auditing configuration items and components. For the purposes of this report, we identified those practices that are embodied, recognized, and accepted acquisition models or frameworks, as well as those practices that are now being recognized as being unique to commercial component-based systems and for which there appears to be general agreement, including agreement with DOD officials responsible for revising the 5000 series, that the practices are relevant and important. Last, we analyzed each of the DOD 5000 series documents to determine whether the documents addressed, either directly or indirectly by reference to another authoritative document, the 18 best practices that we identified. Based on this analysis, we judged whether the 5000 series documents fully, partially, or did not incorporate each best practice. In making these judgments, we used the following criteria: To fully incorporate the practice, the 5000 series addressed all of the practice’s activities.

To partially incorporate the practice, the 5000 series addressed some, but not all, of the practice’s activities.

To not incorporate the practice, the 5000 series did not address any of the practice’s activities.

Additionally, we provided the DOD officials responsible for revising the 5000 series with the 18 practices that we identified to obtain their views on whether the practices were relevant to DOD business systems acquisitions. The officials agreed that they were. We also requested that these officials perform their own assessment of the 5000 series against these practices, and we used these officials’ assessment in making our judgments as to whether the practices were fully, partially, or not incorporated into DOD’s acquisition policies and guidance. For a number of the activities, DOD identified the Federal Acquisition Regulation and the Defense Federal Acquisition Regulation Supplement as evidence that the activity was being performed within a particular practice. We accepted that as proof that the activity was being covered within DOD’s business systems acquisition policy.

To address our second objective, we researched federal internal control standards and controls inherent in the business systems acquisition best practices that we identified. In particular, we reviewed the Software Engineering Institute’s Software Acquisition Capability Maturity Model framework and GAO’s internal control standards. We then analyzed DOD’s revised acquisition policies and guidance to identify whether these controls were cited and to provide assurance that relevant best practices were being followed. We also interviewed DOD officials responsible for revising the 5000 series to determine reasons why controls were addressed or not addressed in the policies and guidance.

Best Practices

Additional information on each of the 18 best practices that we identified is provided in this appendix.

Best Practices Relevant to Any IT Business Systems Acquisition

Purpose: To ensure that reasonable planning for all parts of the acquisition is conducted.

Description: Acquisition planning is the process for conducting and documenting acquisition planning activities beginning early and covering all parts of the project. It extends to all acquisition areas, such as budgeting, scheduling, resource estimating, risk identification, and requirements definition, as well as the overall acquisition strategy. Acquisition planning begins with the earliest identification of a requirement that is to be satisfied through an acquisition.

Activities: (1) Plans are prepared during acquisition planning and maintained throughout the acquisition. (2) Planning addresses the entire acquisition process, including life cycle support of the products being acquired. (3) The acquisition organization has a written policy for planning the acquisition. (4) Responsibility for acquisition planning activities is designated.

Purpose: To ensure that the acquisition is consistent with the organization’s enterprise architecture.

Description: Architectural alignment is the process for analyzing and verifying that the proposed architecture of the system being acquired is consistent with the enterprise architecture for the organization acquiring the system. Such alignment is needed to ensure that acquired systems can interoperate and are not unnecessarily duplicative of one another. Exceptions to this alignment requirement are permitted, but only when justified and only when granted an explicit waiver from the architecture. A particular architectural consideration is whether requirements that extend beyond the specific system being acquired should be considered when selecting system components. Such product line (i.e., systems that are developed from a common set of assets and share a common and managed set of features) considerations can provide substantial production economies over acquiring systems from scratch.

Activities: (1) The system being acquired is assessed for alignment with the enterprise architecture at key life cycle decision points, and any deviations from the architecture are explicitly understood and justified by an explicit waiver to the architecture. (2) Product line requirements— rather than just the requirements for the system being acquired—are an explicit consideration in each acquisition. 3. Contract Tracking and Oversight Purpose: To ensure that contract activities are performed in accordance with contractual requirements.

Description: Contract tracking and oversight is the process by which contractual agreements are established and contractor efforts to satisfy those agreements are supervised. It involves information sharing between the acquirer and contractor to ensure that contractual requirements are understood, that there are regular measurements to disclose overall project status and whether problems exist, and that there are appropriate incentives for ensuring that cost and schedule commitments are met and that quality products are delivered. Contract tracking and oversight begins with the award of the contract and ends at the conclusion of the contract’s period of performance.

Activities: (1) The acquiring organization has sufficient insight into the contractor’s activities to manage and control the contractor and ensure that contract requirements are met. (2) The acquiring organization and contractor maintain ongoing communication; commitments are agreed to and implemented by both parties. (3) All contract changes are managed throughout the life of the contract. (4) The acquiring organization has a written policy for contract tracking and oversight. (5) Responsibility for contract tracking and oversight activities is designated. (6) The acquiring organization involves contracting specialists in the execution of the contract. (7) A quantitative set of software and system metrics is used to define and measure product quality and contractor performance. (8) In addition to incentives for meeting cost and schedule estimates, measurable, metrics-based product quality incentives are explicitly cited in the contract.

Purpose: To ensure that system investments have an adequate economic justification.

Description: Economic justification is the process for ensuring that acquisition decisions are based on reliable analyses of the proposed investment’s likely costs versus benefits over its useful life, as well as an analysis of the risks associated with actually realizing the acquisition’s forecasted benefits for its estimated costs. Moreover, it entails minimizing the risk and uncertainty of large acquisitions that require spending large sums of money over many years by breaking the acquisition into smaller, incremental acquisitions. Economic justification is not a one-time event, but rather is performed throughout an acquisition’s life cycle in order to permit informed investment decision making.

Activities: (1) System investment decisions are made on the basis of reliable analyses of estimated system life cycle costs, expected benefits, and anticipated risks. (2) Large systems acquisitions are (to the maximum extent practical) divided into a series of smaller, incremental acquisition efforts, and investment decisions on these smaller efforts are made on the basis of reliable analyses of estimated costs, expected benefits, and anticipated risks.

Purpose: To ensure that evidence showing that the contract products satisfy the defined requirements are provided prior to accepting contractor products.

Description: Evaluation is the process by which contract deliverables are analyzed to determine whether they meet contract requirements. It includes developing criteria such as product acceptance criteria to be included into both the solicitation package and the contract. It should be conducted continuously throughout the contract period as products are delivered. It begins with development of the products’ requirements and ends when the acquisition is completed.

Activities: (1) Evaluation requirements are developed in conjunction with the contractual requirements and are maintained over the life of the acquisition. (2) Evaluations are planned and conducted throughout the total acquisition period to provide an integrated approach that satisfies evaluation requirements and takes advantage of all evaluation results. (3) Evaluations provide an objective basis to support the product acceptance decision. (4) The acquisition organization has a written policy for managing the evaluation of the acquired products. (5) Responsibility for evaluation activities is designated.

Purpose: To ensure that the project office and its supporting organizations function efficiently and effectively.

Description: Project management is the process for planning, organizing, staffing, directing, and managing all project-office-related activities, such as defining project tasks, estimating and securing resources, scheduling activities and tasks, training, and accepting products. Project management begins when the project office is formed and ends when the acquisition is completed.

Activities: (1) Project management activities are planned, organized, controlled, and communicated. (2) The performance, cost, and schedule of the acquisition are continually measured, compared with planned objectives, and controlled. (3) Problems discovered during the acquisition are managed and controlled. (4) The acquisition organization has a written policy for project management. (5) Responsibility for project management is designated. 7. Requirements Development and Management Purpose: To ensure that contractual requirements are clearly defined and understood by the acquisition stakeholders.

Description: Requirements development is the process for developing and documenting contractual requirements, including evaluating opportunities for reusing existing assets. It involves participation from end users to ensure that product requirements are well understood, and that optional versus mandatory requirements are clearly delineated. Requirements management is the process for establishing and maintaining agreement on the contractual requirements among the various stakeholders and for ensuring that the requirements are traceable, verifiable, and controlled. This involves baselining the requirements and controlling subsequent requirements changes. Requirements development and management begins when the solicitation’s requirements are documented and ends when system responsibility is transferred to the support organization.

Activities: (1) Contractual requirements are developed, managed, and maintained. (2) The end user and other affected groups have input into the contractual requirements over the life of the acquisition. (3) Contractual requirements are traceable and verifiable. (4) The contractual requirements baseline is established prior to release of the solicitation package. (5) The acquisition organization has a written policy for establishing and managing the contractual requirements. (6) Responsibility for requirements development and management is designated. (7) Requirements that are mandatory versus optional are clearly delineated and used in deciding what requirements can be eliminated or postponed to meet other project goals, such as cost and schedule constraints.

Purpose: To ensure that risks are identified and systematically mitigated.

Description: Risk management is the process for identifying potential acquisition problems and taking appropriate steps to avoid their becoming actual problems. It includes risk identification and categorization based on estimated impact, development of risk mitigation strategies, and execution of and reporting on the strategies. Risk management occurs early and continuously in the acquisition life cycle.

Activities: (1) Projectwide participation in the identification and mitigation of risks is encouraged. (2) The defined acquisition process provides for the identification, analysis, and mitigation of risks. (3) Milestone reviews include the status of identified risks. (4) The acquisition organization has a written policy for managing acquisition risk. (5) Responsibility for acquisition risk management activities is designated.

Purpose: To ensure that a quality solicitation is produced and a best- qualified contractor selected.

Description: Solicitation is the process for developing, documenting, and issuing the solicitation package; developing and implementing a plan to evaluate responses; conducting contract negotiations; and awarding the contract. Solicitation ends with contract award.

Activities: (1) The solicitation package includes the contractual requirements and the proposal evaluation criteria. (2) The technical and management elements of proposals are evaluated to ensure that the requirements of the contract will be satisfied. (3) The selection official selects a supplier who is qualified to satisfy the contract’s requirements. (4) The acquiring organization has a written policy for conducting the solicitation. (5) Responsibility for the solicitation is designated. (6) A selection official has been designated to be responsible for the selection process and decision. (7) The acquiring team includes contracting specialists to support contract administration. 10. Transition to Support Purpose: To ensure proper transfer of the system from the acquisition organization to the eventual support organization.

Description: Transition to support is the process for developing and implementing the plans for transitioning products to the support organization. This includes engaging relevant stakeholders in the acquisition and sharing information about the system’s supporting infrastructure. Transition to support begins with requirements development and ends when the responsibility for the products is turned over to the support organization.

Activities: (1) The acquiring organization ensures that the support organization has the capacity and capability to provide the required support. (2) There is no loss in continuity of support to the products during transition from the supplier to the support organization. (3) Configuration management of the products is maintained throughout the transition. (4) The acquiring organization has a written policy for transitioning products to the support organization. (5) The acquiring organization ensures that the support organization is involved in planning for transition to support. (6) Responsibility for transition to support activities is designated.

Complementary Best Practices Relevant to Commercial Component-Based IT Business Systems Acquisitions

Purpose: To ensure that commercial product modification is effectively controlled.

Description: Component modification is the process for limiting the chances of a commercial product being modified to the point that it becomes a one-of-a-kind solution because doing so can result in extensive life cycle costs. Such modifications, if not incorporated into the commercially available version of the product by the supplier, mean that every product release has to be modified in accordance with the custom changes, thus precluding realization of some of the benefit of using a commercial product.

Activity: (1) Modification of commercial components is discouraged and allowed only if justified by a thorough analysis of life cycle costs and benefits.

Purpose: To ensure the integrity and consistency of system commercial components.

Description: Configuration management relative to commercial component-based systems is the process for ensuring that changes to the commercial components of a system are strictly controlled. It recognizes that when using commercial components, it is the vendor, not the acquisition or support organization, that controls the release of new component versions and that new versions are released frequently. Thus, acquisition management needs to provide for both receiving new product releases and controlling the implementation of these releases.

Activities: (1) Project plans explicitly provide for evaluation, acquisition, and implementation of new, often frequent, product releases. (2) Modification or upgrades to deployed versions of system components are centrally controlled, and unilateral user release changes are precluded.

Purpose: To ensure that relationships between commercial products are understood before acquisition decisions are made.

Description: Dependency analysis relative to commercial component- based systems is the process for determining and understanding the characteristics of these products so that inherent dependencies among them can be considered before they are acquired. It involves recognizing that the logical and physical relationships among products impact one another. This is necessary because commercial products are built around each vendor’s functional and architectural assumptions and paradigms, such as approaches to error handling and data access, and these assumptions and paradigms are likely to be different among products from different sources. Such differences complicate product integration. Further, some commercial products have built-in dependencies with other products that, if not known, can further complicate integration.

Activity: (1) Decisions about the acquisition of commercial components are based on deliberate and thorough research, analysis, and evaluation of the components’ interdependencies.4. Legacy Systems Integration Planning Purpose: To ensure reasonable planning for integration of commercial products with existing systems.

Description: Legacy systems integration planning is the process for ensuring that the time and resources needed to integrate existing systems with the system being acquired are identified and provided for. It involves identifying which legacy systems will interact with the system being acquired and what kinds and levels of testing are required. Integration planning recognizes that, although some commercial products may provide mechanisms and information that is helpful in integration with legacy systems, the unavailability of the source code for commercial products and the different organizations that are responsible for the two will likely require additional time and effort.

Activity: (1) Project plans explicitly provide for the necessary time and resources for integrating commercial components with legacy systems. 5. Organization Change Management Purpose: To ensure that the organizational impact of using new system functionality is proactively managed.

Description: Organization change management relative to commercial component-based systems is the process for preparing system users for the business process changes that will accompany implementation of the system. It involves engaging users and communicating the nature of anticipated changes to system users through training on how jobs will change. This is necessary because commercial products are created with the developers’ expectations of how they will be used, and the products’ functionality may require the organization implementing the system to change existing business processes.

Activities: (1) Project plans explicitly provide for preparing users on the impact that the business processes embedded in the commercial components will have on the user’s respective roles and responsibilities. (2) The introduction and adoption of changes to how users will be expected to execute their jobs are actively managed.

Purpose: To ensure that a quality solicitation is produced and a best- qualified contractor is selected.

Description: Solicitation relative to commercial component-based systems is the process for ensuring that a capable contractor is selected. It involves ensuring that the selected contractor has experience with integrating commercial component products. This is important because expertise in developing custom system solutions is different from expertise in implementing commercial components; it requires different core competencies and experiences to be successful.

Activity: (1) Systems integration contractors are explicitly evaluated on their ability to implement commercial components.

Purpose: To ensure that system requirements alone do not drive the system solution.

Description: Tradeoff analysis relative to commercial product-based systems is the process for analyzing and understanding the tradeoffs among competing acquisition variables so as to produce informed acquisition decision making. It involves planning and executing acquisitions in a manner that recognizes four competing interests: defined system requirements, the architectural environment (current and future) in which the system needs to operate, acquisition cost and schedule constraints, and the availability of products in the commercial marketplace (current and future). This analysis should be performed early and continuously throughout an acquisition’s life cycle.

Activity: (1) Investment decisions throughout a system’s life cycle are based on tradeoffs among the availability of commercial products (current and future), the architectural environment in which the system is to operate (current and future), defined system requirements, and acquisition cost/schedule constraints.8. Vendor and Product Research and Evaluation Purpose: To ensure that vendor and product characteristics are understood before acquisition decisions are made.

Description: Vendor and product research and evaluation relative to commercial component-based systems is the process for obtaining reliable information about both the product being considered and the vendor offering the product. It involves taking additional steps beyond vendor representations, such as obtaining information about the vendor’s history, obtaining information on the vendor’s business strategy relative to evolution and support of the product, and evaluating copies of the product in a test environment.

Activities: (1) Commercial component and vendor options are researched, evaluated/tested, and understood, both early and continuously. (2) A set of evaluation criteria for selecting among commercial component options is established that includes both defined system requirements and vendor/commercial product characteristics (e.g., customer satisfaction with company and product line).

Comments from the Department of Defense

The following are GAO’s comments on the Department of Defense’s letter dated July 1, 2004.

GAO Comments

1. We disagree. While we have updated our report to reflect the additional information provided in DOD’s comments on the status of its efforts and the associated milestone, the importance of revising and maintaining DOD's acquisition policies and guidance, and their incorporation of acquisition best practices, makes it essential to have an explicit plan. Among other things, a plan would highlight the resource constraints that this revision effort has been subject to, would allow measurement against defined milestones, and would allow disclosure of progress and impediments. 2. See comment 1. 3. We do not question DOD’s statement that it has an initial repository of custom software components that can help adapt commercial components to defense use and reuse. However, this repository does not satisfy the product line requirements in this best practice and our recommendation. According to the Software Engineering Institute, the product line requirements best practice involves more than just reuse. Under the approach described in DOD’s comments, reuse generally involves items, such as software modules or components, that developers are encouraged to use. However, the product line requirements best practice is not simply encouraging reuse of items in a repository. Rather, it is planned, enabled, and enforced reuse of such assets as requirements, models, and architectures that have been designed and optimized for use in multiple systems. In short, it is proactive rather than reactive reuse. 4. We have modified our recommendation to use the terminology “acquisition project management activities” instead of “acquisition management activities” to eliminate any confusion. Further, we do not question DOD’s use of integrated product teams as a way to communicate information. Further, the point of our recommendation is that there needs to be an explicit recognition in policy and guidance of the type of information to be communicated and to whom. Incorporating this best practice is based on the need to ensure that communication vehicles, such as integrated product teams, are effective. 5. We agree that the 5000 series contains information on acquisition risk management, as we state in our report, and that one of the purposes of DOD’s acquisition framework is to reduce risks. However, there is no provision in DOD’s acquisition policy or guidelines to ensure that the status of identified risks are discussed at key decision points. For example, DOD policy states that a criterion for passing milestone A is “management and mitigation of technology risk.” However, it does not provide for what is to be done to manage and mitigate risks, and it does not provide for reviewing risk status at milestones B or C. 6. While we agree that the toolkit provides relevant change management information, the toolkit is referenced only once in the 5000 series; and the reference is only in relation to one phase of the acquisition cycle (the technology development phase). Given the importance and relevance of this practice to the successful implementation of commercial component-based systems, our position, and thus the basis for our recommendation, is that the best practice’s implementation would more likely occur if it was visible and recognized in all relevant stages of DOD’s acquisition cycle. 7.

See comment 6. 8. While the regulations and guidance that DOD cited and referenced in its acquisition policies discuss the use of information on contractors’ past performance, they do not discuss evaluating systems integration contractors on their ability to implement commercial components, which is the point of the best practice. Further, DOD’s objection to incorporating this best practice is not consistent with its own comments. Specifically, DOD commented that it has already taken steps, through its enterprise systems initiative, to establish blanket agreements with five contractors who were evaluated on, among other things, “explicitly proven abilities to implement commercial systems solutions.” Additionally, while we appreciate DOD’s concern that incorporation of this best practice can have unintended consequences, it is important to also recognize that our recommendation is not intended to restrict a contracting officer’s options. Rather, it is intended to be one of the many factors considered in the source selection process when it is relevant. 9. We support the Defense Acquisition Executive’s decision to add more specifics on systems engineering to the 5000 series, including provisions that address this best practice and recommendation. 10. See comment 8. 11. We disagree. While we do not question that statutory and regulatory compliance are referenced in DOD’s integrated process team and milestone decision point processes, we do not believe that these reviews are adequately defined with respect to implementation of best practices because DOD’s policy does not require that the practices’ use be measured and verified. Rather, it leaves these reviews to the discretion of the program manager and investment decision authority. As we state in our report, not requiring that the use of best practices be measured and verified increases the chance that the practices will not be followed. Therefore, our position remains that DOD’s policies do not provide effective controls for ensuring that best practices are appropriately followed.

GAO Contact and Staff Acknowledgments

Carl L. Higginbotham, (404) 679-1824 Staff acknowledgments In addition to the individual named above, key contributors to this report included Nabajyoti Barkakati, Nancy Glover, Madhav Panwar, Morgan Walts, and Thomas Wright.

GAO’s Mission

The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to www.gao.gov and select “Subscribe to Updates.”

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs