United States General Accounting Office

GAO Report to the Chairman, Subcommittee

on the District of Columbia, Committee

on Appropriations, House of

Representatives

April 1998 DISTRICT OF

COLUMBIA

Software Acquisition

Processes for a New

Financial Management

System

GAO/AIMD-98-88

GAO United States

General Accounting Office

Washington, D.C. 20548

Accounting and Information

Management Division

B-279266 

April 30, 1998

The Honorable Charles H. Taylor

Chairman, Subcommittee on the

 District of Columbia

Committee on Appropriations

House of Representatives

Dear Mr. Chairman:

This report responds to your request that we determine whether the

District of Columbia had implemented disciplined software acquisition

processes for its new financial management system (FMS). We found that

the District’s software acquisition processes for the FMS acquisition, while

having some strengths, are not mature when compared to standards

established by the Software Engineering Institute (SEI).1

 Accordingly, we

are making recommendations for strengthening these processes as they

relate to the FMS project and to improve any future software acquisitions.

Background In September 1997, the District of Columbia Financial Responsibility and

Management Assistance Authority (Authority) awarded a contract to

acquire a new FMS. The overall objective of the FMS project is to improve

the District’s financial systems through faster, more efficient, and accurate

processing providing increased functionality, flexibility, and reduced cost

of operations. According to the Chair of the Authority, the new FMS is

intended to (1) eliminate the principal problems that exist with the current

system and ensure that all financial management guidelines are adhered

to, (2) enable managers to more effectively and efficiently monitor and

control financial resources, and (3) produce timely, accurate, and reliable

information, providing decisionmakers with the basic financial

information needed to make more informed decisions.

The Authority awarded a contract for the new FMS in September 1997 and

committed to an aggressive implementation schedule. The schedule

anticipates (1) pilots in five agencies beginning in February 1998, (2) the

accounting system to be implemented by October 1998, and

(3) District-wide implementation by February 1999.

1

Carnegie Mellon University’s Software Engineering Institute (SEI), recognized for its expertise in

software processes, has developed models and methods that define and determine organizations’

software process maturity.

Page 1 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Objective, Scope, and

Methodology

We were asked to review the District’s efforts to acquire a new financial

management system. Our objective was to determine whether the District

had implemented disciplined software acquisition processes for acquiring

its new financial management system.

To accomplish this, we applied the Software Engineering Institute’s

Software Acquisition Capability Maturity Model (SA-CMM) and its Software

Capability Evaluation (SCE) method. SEI’s expertise in, and methods for,

software process assessment are recognized and accepted throughout the

industry. Our evaluators were all SEI-trained software specialists.

SA-CMM ranks organizational maturity according to five levels (see figure 1).

Maturity levels 2 through 5 require the verifiable existence and use of

certain software acquisition processes, known as key process areas (KPA).

According to SEI, an agency that has these acquisition processes in place is

in a much better position to successfully acquire software than an

organization that does not have these processes in place. We evaluated the

District’s software acquisition processes against six of the seven level 2

KPAs (the transition to support KPA was not evaluated because the District

does not plan to support FMS in-house) and one level 3 KPA (see table 1).

We selected level 2 because it is the minimum level at which any

assurance exists that software acquisition processes are mature enough to

consistently deliver promised software capabilities on time and within

budget. We included one level 3 KPA—acquisition risk

management—because it is considered by software experts to be a very

important process area.

Page 2 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Figure 1: SA-CMM Levels and Descriptions

The acquisition organization's software acquisition

process is documented, standardized, and established

as the standard software acquisition process. All 

projects use an approved, tailored version of the

organization's standard software acquisition process

for acquiring their software products and services.

Level 3 - Defined

Detailed measures of quality of the software acquistion

processes, products, and services are collected. The

software processes, products, and services are

quantitatively understood and controlled.

Level 4 - Managed

Continuous process improvement is empowered by

quantitative feedback from the process and from piloting 

innovative ideas and technologies.

Level 5 - Optimizing

Basic project management processes are established

to track performance, cost, and schedule. The

necessary process discipline is in place to repeat

earlier successes on projects in similar domains.

Level 2 - Repeatable

Continuously 

improving

process

Predictable

process

Standard

consistent

process

The software acquisiton process is characterized as 

ad hoc, and occasionally even chaotic. Few processes

are defined and success depends on individual

effort.

Level 1 - Initial

Disciplined

process

Page 3 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Table 1: SA-CMM KPAs Used to

Assess the FMS Project Software

Acquisition Maturity

SA-CMM level 2 key

process areas Description

Software acquisition planning Ensuring that reasonable planning for the software

acquisition is conducted and that all elements of the

project are included.

Solicitation Ensuring that award is made to the contractor most

capable of satisfying the specified requirements.

Requirements development

and management

Establishing a common and unambiguous definition of

software acquisition requirements understood by the

acquisition team, system user, and the contractor.

Project management Managing the activities of the project office and

supporting contractor(s) to ensure a timely, efficient, and

effective software acquisition.

Contract tracking and

oversight

Ensuring that the software activities under contract are

being performed in accordance with contract

requirements, and that products and services will satisfy

contract requirements.

Evaluation Determining that the acquired software products and

services satisfy contract requirements prior to

acceptance and transition to support.

SA-CMM level 3 key

process area Description

Acquisition risk management Identifying risks as early as possible, adjusting acquisition

strategy to mitigate those risks, and developing and

implementing a risk management process as an integral

part of the acquisition process.

As established by the model, each KPA contains five common attributes

that indicate whether the implementation and institutionalization of a KPA

can be effective, repeatable, and lasting. The five common attributes are:

Commitment to perform. Commitment to perform describes the actions

that the organization must take to establish the process and ensure that it

can endure. Commitment to perform typically involves establishing

organizational policies and sponsorship. Key practices under commitment

to perform include having written organization policy for the process area

and designating and assigning responsibility to individuals to perform the

activities.

Ability to perform. Ability to perform describes the preconditions that

must exist in the project or organization to implement the software

acquisition process competently. Ability to perform typically involves

resources, organizational structures, and training. Key practices under

ability to perform include having either experienced or trained personnel

Page 4 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

and providing adequate resources to conduct the activities for the process

area.

Activities performed. Activities performed describes the roles and

procedures necessary to implement a KPA. Activities performed typically

involve establishing plans and procedures, performing the work, tracking

it, and taking appropriate management actions. Key practices under

activities performed include performing these activities in accordance

with documented plans and overall management of the project team and

process area.

Measurement and analysis. Measurement and analysis describes activities

performed to measure the process and analyze the measurements.

Measurement and analysis typically includes defining the measurements to

be taken and the analyses to be conducted to determine the status and

effectiveness of the activities performed.

Verifying implementation. Verifying implementation describes the steps to

ensure that the activities are performed in compliance with the process

that has been established. There are two key practices under verifying

implementation, including reviews by various levels of management.

In accordance with SEI’s methodology, for each KPA selected, we evaluated

the District’s policies and practices. This project-specific comparison can

result in one of four possible outcomes: (1) project strength—an effective

implementation of the key practice, (2) project weakness—ineffective

implementation of a key practice or failure to implement a key practice,

(3) project observation—key practice evaluated but evidence inconclusive

and cannot be characterized as either a strength or weakness, and (4) not

rated—key practice not currently relevant to project, therefore not

evaluated.

As requested, we applied the SEI methodology to evaluate the District’s

management of only one project—the acquisition of a new financial

management system. As a result, we have no information regarding the

process strengths and weaknesses of other acquisitions the District may

have underway.

We performed our work at District of Columbia offices in Washington,

D.C. between September 16, 1997, and January 16, 1998, in accordance

with generally accepted government auditing standards.

Page 5 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Results in Brief While the District has many strengths in its acquisition processes for FMS,

it also has many weaknesses. When compared to standards established by

the SEI, the District’s processes for software acquisition are not mature. Of

the six KPAs evaluated for the repeatable level, the District fully satisfied

only one—solicitation. Severe weaknesses were found in other critical key

processes, including requirements development and management and

evaluation. For example, the District does not have a policy for

establishing and managing software-related requirements, does not, at

present, have adequate resources for requirements development, and has

not formally designated responsibility for requirements development and

management.

Similarly, the District does not have an effective evaluation process, and is

currently unable to objectively determine if the acquired system will

satisfy the contract requirements. Finally, the District has not satisfied the

one key process area evaluated for the “defined” level of maturity,

acquisition risk management. The FMS project does not have a risk

management plan and does not track project risk. Figure 2 provides a

comprehensive listing of the FMS project’s strengths, weaknesses, and

observations for the seven KPAs. The items under features in figure 2 refer

to the five common attributes in each KPA and are explained in detail in the

following sections.

Page 6 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Figure 2: Key Process Area Results

Requirements

development

and

management

Feature

Software

acquisition

planning

Solicitation Project

management

Contract

tracking and

oversight

Evaluation Acquisition risk

management

Commitment 1 Weakness Strength Weakness Weakness Weakness Weakness

Weakness Weakness

Weakness Weakness

Weakness Weakness Weakness

Commitment 2

Commitment 3

Ability 1

Ability 2

Ability 3

Ability 4

Activity 1

Activity 2

Activity 3

Activity 4

Activity 5

Activity 6

Measurement 1

Verification 1

Verification 2

Weakness

Weakness

Weakness Weakness Weakness Weakness Weakness Weakness Weakness

Weakness

Weakness Weakness

Weakness

Weakness

Weakness Weakness Weakness

Weakness

Weakness

Weakness

Weakness

Weakness

Weakness

Weakness

Weakness

Strength Strength Strength Strength Strength

Strength Strength Strength Strength Strength

Strength

Strength Strength Strength

Not rated Not rated Not rated

Strength Strength Strength Strength

Strength Strength Strength Strength

Strength Strength Strength Strength

Strength Strength

Strength Strength Strength Strength

Strength

Strength

Strength

Strength

Strength

Strength

Strength

Strength

Strength

Strength

Strength

Not rated Strength Not rated Not rated Not rated Not rated

Not rated

Not rated Strength Strength Strength

Observation

Observation

Observation Not rated Weakness

Not rated Not rated Not rated Observation Not rated

Observation

Observation Observation Observation

Weakness

Weakness

Weakness

= Weakness

= Strength

= Observation

Key practice not implemented

Key practice effectively implemented

Key practice evaluated but evidence inconclusive. Cannot characterize as either strength

or weakness

= Not rated Key practice not currently relevant to project, therefore not evaluated

Page 7 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Software Acquisition

Planning

The purpose of software acquisition planning is to ensure that reasonable

planning for the software acquisition is conducted and that all aspects of

the total software acquisition effort are included in these plans at the

proper level of detail. The software acquisition planning process, among

other things, includes (1) addressing software life-cycle support in

acquisition plans, (2) preparing life-cycle software cost estimates,

(3) having a written software acquisition policy, (4) measuring and

reporting on the status of software acquisition planning activities, and

(5) having guidance on software training and experience requirements for

project personnel.

FMS Project Performs

Some but Not All Software

Acquisition Planning

Practices

The FMS project had many strengths in this KPA. The District received pro

bono assistance from several companies to help define the acquisition

strategy and conduct the activities for software acquisition planning. An

acquisition strategy was developed and the acquisition planning team was

staffed with personnel with software and systems experience. The team

developed a cost estimate and the District management was briefed by the

team on a periodic basis. This enabled the District management to be

informed on the progress of the acquisition planning and the various

activities through the solicitation phase.

However, the FMS project also had many weaknesses in this KPA.

Weaknesses observed included a lack of policy on acquisition planning

and no specific assignment of responsibility for acquisition planning.

Furthermore, the FMS project did not always document significant project

decisions or update the planning document to reflect these decisions. For

example, when the District decided to not pursue a single contract to both

acquire FMS and outsource data center operations, the capability

assessment (a software acquisition planning document) was not updated

to reflect this decision. Decisions should be documented and the planning

documents updated to ensure that large acquisitions such as FMS can be

effectively managed. Table 2 shows the strengths and weaknesses for the

software acquisition planning KPA and the specific findings supporting

these ratings.

Page 8 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Table 2: Software Acquisition Planning Findings

Feature Key practice Finding Rating

Commitment 1 The acquisition organization has a written policy

for planning the software acquisition.

There is no written policy for software

acquisition planning.

Weakness

Commitment 2 Responsibility for software acquisition planning

activities is designated.

Responsibility for software acquisition planning

activities was not designated.

Weakness

Ability 1 The acquisition organization has experienced

software acquisition management personnel.

The acquisition staff (10-15 Authority and D.C.

government staff, and 10-15 private sector

employees working on a pro bono basis) had

requisite experience.

Strength

Ability 2 Adequate resources are provided for software

acquisition planning activities.

Required resources for acquisition planning

were not determined.

Weakness

Activity 1 Software acquisition planning personnel are

involved in system acquisition planning.

The District government recruited outside

software experts to assist with system

acquisition activities.

Strength

Activity 2 The software acquisition strategy for the project

is developed and documented.

The software acquisition strategy is developed

and includes risk identification and schedules. It

is documented in the capability assessment

document.

Strength

Activity 3 The project’s software acquisition planning is

documented and the planning documentation is

maintained over the life of the project.

Program changes regarding outsourcing of the

data center and upgrading the current system

versus buying off the shelf were made, but the

software acquisition planning documentation

was not updated.

Weakness

Activity 4 Life-cycle support of the software is included in

software acquisition planning documentation.

The software acquisition planning

documentation does not address life cycle

support of the software.

Weakness

Activity 5 Life-cycle cost and schedule estimates for the

software products and services being acquired

are prepared and independently reviewed.

Life-cycle cost and schedule estimates were

prepared by a consultant. They were reviewed

by the District Chief Financial Officer and

Authority Executive Director.

Strength

Measurement 1 Measurements are made and used to determine

the status of the software acquisition planning

activities and resultant products.

Measurements of software acquisition activities

were not taken.

Weakness

Verification 1 Software acquisition planning activities are

reviewed by acquisition organization

management on a periodic basis.

The Authority Executive Director was briefed on

planning activities on a periodic basis.

Strength

Verification 2 Software acquisition planning activities are

reviewed by the project manager on both a

periodic and event-driven basis.

Software acquisition planning activities were

reviewed by the project manager.

Strength

Solicitation The purpose of solicitation is to prepare a request for proposal that

delineates a project’s software-related requirements and select a

contractor that can most cost-effectively satisfy these requirements while

complying with relevant solicitation laws and regulations. Specific

Page 9 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

requirements for a solicitation process include, among other things

(1) having and following a solicitation plan, (2) assigning responsibility

and ensuring sufficient resources for coordinating and conducting

solicitation activities, (3) preparing and reviewing cost and schedule

estimates for the software products and services being acquired, and

(4) periodically measuring solicitation work completed and effort and

funds expended, comparing these measures to plans, and reporting the

results to management.

FMS Project Performing

Most Solicitation Practices

The FMS project exhibited many process strengths during the solicitation.

The District has a policy on solicitation and the FMS project followed this

policy. The project had experienced personnel on the source selection

team and these personnel briefed the team members on the objectives of

the solicitation. However, the District did not measure either time or funds

expended to conduct the solicitation. Specifically, no evidence was

provided to show that the FMS project tracked personnel hours or costs

during the conduct of the solicitation. Addressing this weakness would

enable the District to better estimate the resources needed to conduct

similar acquisitions in the future. For example, if these data were collected

and made available to other projects, such as the tax systems upgrade, the

District would be in a better position to understand its own capability to

effectively conduct solicitation, to estimate how long such a solicitation

was likely to take, and to eliminate problems that may have hampered the

FMS solicitation. Table 3 shows the strengths, weaknesses, and

observations for the solicitation KPA and the specific findings supporting

these ratings.

Table 3: Solicitation Findings

Feature Key practice Finding Rating

Commitment 1 The acquisition organization has a written policy

for the conduct of the software portion of the

solicitation.

Authority’s procurement regulations provides

written policies for solicitation.

Strength

Commitment 2 Responsibility for the software portion of the

solicitation is designated.

Authority’s Executive Director has designated

specific individuals to serve as members of a

selection committee.

Strength

Commitment 3 A selection official has been designated to be

responsible for the selection process and the

decision.

Authority’s Executive Director is designated as

the source selection official.

Strength

Ability 1 A group that is responsible for coordinating and

conducting solicitation activities exists.

A solicitation organization has been identified in

the source selection plan.

Strength

(continued)

Page 10 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Feature Key practice Finding Rating

Ability 2 Adequate resources are provided for solicitation

activities.

Adequate resources (including Authority staff

and a team of private sector experts) were

provided for solicitation activities.

Strength

Ability 3 Individuals performing solicitation activities have

experience or receive training.

Experienced individuals from private contractor

staff and public sectors assisted in the

solicitation activities.

Strength

Ability 4 The groups supporting the solicitation (e.g., end

user, systems engineering, software support

organization, and application domain experts)

receive orientation on the solicitation’s

objectives and procedures.

Participants in source selection process were

trained on the objectives and procedures of

FMS solicitation.

Strength

Activity 1 The project team performs its activities in

accordance with its documented solicitation

plans.

Project team conducted its activities according

to the source selection plan, however, that plan

does not cover all solicitation tasks.

Observation

Activity 2 The project team performs its activities in

accordance with its documented proposal

evaluation plans.

A number of subcommittees evaluated

proposals according to the process

documented in the source selection plan.

Strength

Activity 3 Cost and schedule estimates for the software

products and services being sought are

prepared.

Capabilities assessment report documents the

cost and schedule estimates for the new

financial management system. A preliminary

economic analysis of FMS was also performed.

Strength

Activity 4 Software cost and schedule estimates are

independently reviewed for comprehensiveness

and realism.

Capabilities assessment report and the

economic analysis of FMS were reviewed by the

D.C. Office of the Inspector General and a

report was issued.

Strength

Activity 5 The project team takes action to ensure the

mutual understanding of software requirements

and plans prior to contract award.

The project team ensured mutual understanding

of software requirements by holding question

and answer sessions with potential FMS

vendors.

Strength

Measurement 1 Measurements are made and used to determine

the status of the solicitation activities and

resultant products.

No measurements were taken to determine the

status of solicitation activities.

Weakness

Verification 1 Solicitation activities are reviewed by the

designated selection official or acquisition

organization management on a periodic basis.

Authority Executive Director and the District

Chief Financial Officer are briefed on status of

all FMS activities (including solicitation) on a

periodic basis.

Strength

Verification 2 Solicitation activities are reviewed by the project

manager on both a periodic and event-driven

basis.

During the solicitation, the entire team reported

to the Executive Director who served as the

project manager. The Executive Director was

briefed periodically and in response to

significant events.

Strength

Requirements

Development and

Management

The purpose of requirements development and management is to establish

and maintain a common and unambiguous definition of software

requirements among the acquisition team, system users, and software

development contractor. This KPA involves two subprocesses:

Page 11 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

(1) developing a baseline set of software-related contractual requirements

and (2) managing these requirements and changes to these requirements

for the duration of the acquisition.

A number of requirements development and management practices are

necessary to satisfy this key process area. These include (1) having a

written organizational policy for establishing and managing requirements

allocated to software, (2) documenting plans for the development and

management of requirements, (3) having documented processes for

requirements development, including elicitation, analysis, and verification,

(4) measuring and reporting on the status of requirements development

and management activities to management, (5) appraising the impact on

software of system-level requirements changes and (6) having a

mechanism to ensure that contractor-delivered work products meet

specified requirements.

FMS Project Performing

Some but Not Most

Requirements

Development and

Management Practices

The FMS project has some process strengths in the conduct of

requirements development and management. The project team is

performing requirements management activities in accordance with its

documented plan and software-related contractual requirements have

been baselined. In addition, District management periodically reviews the

status of requirements development and management activities with the

project team. However, in acquiring FMS, the District did not perform many

of the requirements development and management practices necessary to

satisfy this KPA. For example, the District does not have an organizational

policy for establishing and managing software-related requirements, there

is no clear assignment of responsibility for requirements development and

management and no documented evidence exists to show either resource

requirements or resources expended for requirements development

activities.

Currently, the FMS project has begun to hold “requirements confirmation

meetings” with the users to validate the requirements already specified in

the FMS contract. Although requirements should be validated, this should

have been done prior to releasing the request for proposal to ensure that

the proposal accurately reflects the District’s requirements. Changing

requirements after contract award may adversely impact project cost,

schedule, and/or performance. Table 4 shows the strengths, weaknesses,

and observations for the requirements development and management KPA

and the specific findings supporting these ratings.

Page 12 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Table 4: Requirements Development and Management Findings

Feature Key practice Finding Rating

Commitment 1 The acquisition organization has a written policy

for establishing and managing the

software-related contractual requirements.

There is no organizational policy for establishing

and managing software-related requirements.

Weakness

Commitment 2 Responsibility for requirements development

and management is designated.

There is no formal documentation assigning

anyone responsibility for requirements

development and management.

Weakness

Ability 1 A group that is responsible for performing

requirements development and management

activities exists.

There is no documentation assigning any group

responsibility for performing requirements

development and management activities.

Weakness

Ability 2 Adequate resources are provided for

requirements development and management

activities.

No documented evidence exists to show either

resource requirements or resources used for

requirements development and management

activities.

Weakness

Ability 3 Individuals performing requirements

development and management activities have

experience or receive training.

Individuals currently performing requirements

management have appropriate experience.

However, no documented evidence was

provided to show the experience of individuals

who developed the requirements.

Observation

Activity 1 The project team performs its activities in

accordance with its documented requirements

development and management plans.

The requirements management activities are

incorporated in the project management plan,

volumes I and II. This project team is performing

its activities in accordance with this plan.

Strength

Activity 2 The project team develops and baselines the

software-related contractual requirements and

places them under change control early in the

project, but not later than release of the

solicitation package.

The project team baselined software-related

contractual requirements in the statement of

work prior to the release of the solicitation. The

requirements are under change control.

Strength

Activity 3 The project team appraises system

requirements change requests for their impact

on the software being acquired.

There have been no requirements changes to

date to appraise. However, a process to

appraise changes exists and is documented in

the program management plan.

Observation

Activity 4 The project team appraises all changes to the

software-related contractual requirements for

their impact on performance, architecture,

supportability, system resource utilization, and

contract schedule and cost.

The scope change process documented in the

project management plan requires the appraisal

of cost and schedule impact. However, no

evidence provided of any changes to the

contract to date.

Observation

Activity 5 Bi-directional traceability between the

software-related contractual requirements and

the contractor’s software work products and

services is maintained throughout the effort.

There is no evidence to show traceability

between contractual requirements and the

contractor’s work products.

Weakness

Measurement 1 Measurements are made and used to determine

the status of the requirements development and

management activities and resultant products.

No measurements made to determine status of

the requirements development and

management activities.

Weakness

(continued)

Page 13 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Feature Key practice Finding Rating

Verification 1 Requirements development and management

activities are reviewed by acquisition

organization management (and the contractor)

on a periodic basis.

Project manager periodically reviews

requirements development and management

activities.

Strength

Verification 2 Requirements development and management

activities are reviewed by the project manager

on both a periodic and event-driven basis.

Periodic and event-driven meetings are held to

discuss requirements development and

management activities.

Strength

Project Management The purpose of project management is to manage the activities of the

project office and supporting contractors to ensure a timely, efficient, and

effective software acquisition. Effective project management requires,

among other things, that project teams (1) be organized to accomplish the

project’s objective, (2) have a written policy for the management of the

software project, (3) document their plans for the activities of the project

team, (4) have the authority to alter either the project’s performance, cost,

or schedule baseline while maintaining the other two, and (5) periodically

brief management on the status of project management activities.

FMS Project Performing

Most Project Management

Practices

The FMS project has many process strengths in project management. For

example, a team was assigned responsibility for managing the project and

staffed with experienced individuals whose roles and responsibilities were

defined. The program management plan was written and a corrective

action system to track issues and problems was implemented. However,

the District has no written policy for the execution of the software project.

As a result, the District has no assurance that FMS or any other software

acquisition project it undertakes will be conducted in a disciplined

manner. Table 5 shows the strengths, weaknesses, and observations for

the project management KPA and the specific findings supporting these

ratings.

Page 14 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Table 5: Project Management Findings

Feature Key practice Finding Rating

Commitment 1 The acquisition organization has a written policy

for execution of the software project.

There is no documented policy for the execution

of the software project.

Weakness

Commitment 2 Responsibility for project management is

designated.

Responsibility for project management has

been designated to the financial management

system project team/program manager.

Strength

Ability 1 A team that is responsible for performing the

project’s software acquisition management

activities exists.

A team that is responsible for performing

software acquisition management activities

exists.

Strength

Ability 2 Adequate resources for the project team and

matrix support persons are provided for the

duration of the software acquisition project.

Adequate personnel resources for project

management were provided. At the time of the

audit, there were no telephones, copiers, or

supplies.

Observation

Ability 3 When project trade-offs are necessary, the

project manager is permitted to alter either the

performance, cost, or schedule software

acquisition baseline.

The project manager is not permitted to

independently alter either the performance,

cost, or schedule.

Weakness

Ability 4 The project team and matrix support

individual(s) have experience or receive training

in project software acquisition management

activities.

Project team and matrix support individuals

have experience in acquisition management

activities.

Strength

Activity 1 The project team performs its activities in

accordance with its documented software

acquisition management plans.

Project team activities are performed in

accordance with program management plan.

Strength

Activity 2 The organization of the project provides for the

management of all project functions.

The organization of the project provides for the

management of all project functions.

Strength

Activity 3 The software acquisition management activities

of the project team are directed to accomplish

the project’s objectives.

The roles and responsibilities of team members

are defined and serve to accomplish the

project’s objectives.

Strength

Activity 4 The software acquisition management activities

of the project team are controlled.

The software acquisition management activities

of the team are controlled.

Strength

Activity 5 The project team implements a corrective action

system for the identification, recording, tracking,

and correction of problems discovered during

the software acquisition.

A corrective action system, called the issue

process, has been implemented.

Strength

Activity 6 The project team tracks project status,

execution, funding, and expenditures and takes

action.

The project team tracks the project’s status and

execution. Funding and expenditures are

tracked by the Office of the Chief Financial

Officer.

Strength

Measurement 1 Measurements are made and used to determine

the status of the project management activities

and resultant products.

No evidence that measurements are taken to

determine the status of project management

activities.

Weakness

Verification 1 Project management activities are reviewed by

acquisition organization management on a

periodic basis.

Monthly meetings are held where all levels of

management are briefed on the project status.

Strength

Verification 2 Project management activities are reviewed by

the project manager on both a periodic and

event-driven basis.

Monthly meetings are held where all levels of

management are briefed on the project status.

Strength

Page 15 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Contract Tracking and

Oversight

The purpose of contract tracking and oversight is to ensure that (1) the

software development contractor performs according to the terms of the

contract, (2) needed contract changes are identified, negotiated, and

incorporated into the contract, and (3) contractor performance issues are

identified early, when they are easier and less costly to address. An

effective contract tracking and oversight process, among other things,

includes (1) having a written organizational policy for contract tracking

and oversight, (2) having a documented plan for contract tracking and

oversight, (3) conducting tracking and oversight activities in accordance

with the plan, and (4) ensuring that individuals performing contract

tracking and oversight are suitably experienced or trained.

FMS Project Performing

Many but Not All Contract

Tracking and Oversight

Practices

The FMS project had many strengths in this KPA. The project has a

designated project manager, a group is responsible for managing contract

tracking and oversight activities, and the team is meeting periodically with

the contractor and tracking issues in a corrective action system. However,

at the time of our review, there was no contracting specialist supporting

the team in the execution of the contract. In addition, the District has no

documented policy for contract tracking and oversight activities. Table 6

shows the strengths, weaknesses, and observations for the contract

tracking and oversight KPA and the specific findings supporting these

ratings.

Table 6: Contract Tracking and Oversight Findings

Feature Key practice Finding Rating

Commitment 1 The acquisition organization has a written policy

for contract tracking and oversight of the

contracted software effort.

There is no written policy for contract tracking

and oversight activities for the financial

management system project.

Weakness

Commitment 2 Responsibility for contract tracking and

oversight activities is designated.

The project manager and the program

management team are responsible for contract

tracking and oversight.

Strength

Commitment 3 The project team is supported by contracting

specialists in the execution of the contract.

The project team is not supported by

contracting specialists.

Weakness

Ability 1 A group that is responsible for managing

contract tracking and oversight activities exists.

Project manager and program management

staff are responsible for managing contract

tracking and oversight activities.

Strength

Ability 2 Adequate resources are provided for contract

tracking and oversight activities.

Staff and funding resources are adequate.

However, at the time of the audit, the project

team was lacking tools such as telephones, fax,

copiers, and office supplies.

Observation

Ability 3 Individuals performing contract tracking and

oversight activities have experience or receive

training.

Staff members have experience in contract

tracking and oversight.

Strength

(continued)

Page 16 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Feature Key practice Finding Rating

Activity 1 The project team performs its activities in

accordance with its documented contract

tracking and oversight plans.

The project team performs its activities in

accordance with the program management plan.

Strength

Activity 2 The project team reviews required contractor

software planning documents which, when

satisfactory, are used to oversee the

contractor’s software engineering effort.

The project team did not review any of the

contractor’s planning documents (e.g., project

management plan, software risk management

plan, software engineering plan, configuration

management plan).

Weakness

Activity 3 The project team conducts periodic reviews and

interchanges with the contractor.

The team conducts periodic reviews with the

contractor.

Strength

Activity 4 The project team reviews and tracks the

development of the software engineering

environment required to provide life-cycle

support for the acquired software.

The District does not plan to support FMS

in-house.

Not rated

Activity 5 Any problems or issues found by the project

team during contract tracking and oversight are

recorded in the appropriate corrective action

system and tracked to closure.

The issue tracking system is used to track and

provide oversight of problems or issues found

during the contract. Issues are tracked to

closure.

Strength

Activity 6 The project team maintains the integrity of the

contract throughout the contract performance

period.

No one assumed responsibility for maintaining

the integrity of the contract.

Weakness

Measurement 1 Measurements are made and used to determine

the status of the contract tracking and oversight

activities and resultant products.

No measurements are taken to determine the

status of contract tracking and oversight

activities.

Weakness

Verification 1 Contract tracking and oversight activities are

reviewed by acquisition organization

management on a periodic basis.

Periodic and event-driven meetings are held

with the Authority to review contract tracking

and oversight activities.

Strength

Verification 2 Contract tracking and oversight activities are

reviewed by the project manager on both a

periodic and event-driven basis.

Periodic and event-driven meetings are held

with the program manager to review contract

tracking and oversight activities.

Strength

Evaluation The purpose of evaluation (testing) is to determine that the acquired

software products and services satisfy contract requirements prior to

acceptance. The evaluation process includes (1) documenting evaluation

plans and conducting evaluation activities in accordance with the plan,

(2) developing and managing evaluation requirements in conjunction with

developing software technical requirements, (3) incorporating evaluation

requirements into the solicitation and the resulting contract, (4) tracking

contractor performance of evaluation activities for compliance with the

contract, (5) ensuring that adequate resources are provided for evaluation

activities, and (6) measuring and reporting on the status of evaluation

activities to management.

Page 17 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

FMS Project Performing

Some but Not Most

Evaluation Practices

The FMS project has some process strengths in this KPA. For example,

responsibility for evaluation activities has been designated to the project

manager, individuals designated to perform evaluation activities have

experience, and members of the evaluation team received briefings on the

objectives of the evaluation. However, there is no documented evaluation

policy or plan, no evidence that evaluation requirements have been

developed, and neither the Authority nor the project manager reviews the

status of evaluation activities. Table 7 shows the strengths, weaknesses,

and observations for the evaluation KPA and the specific findings

supporting these ratings.

Table 7: Evaluation Findings

Feature Key practice Finding Rating

Commitment 1 The acquisition organization has a written policy

for managing the evaluation of the acquired

software products and services.

No written policy exists for managing the

evaluation of acquired software products and

services.

Weakness

Commitment 2 Responsibility for evaluation activities is clearly

designated.

Responsibilities for evaluation activities has

been designated to the program manager.

Strength

Ability 1 A group that is responsible for planning,

managing, and performing evaluation activities

for the project exists.

The joint program management and contractor

team, along with the users, is responsible for

planning, managing, and performing evaluation

activities.

Strength

Ability 2 Adequate resources are provided for evaluation

activities.

While personnel resource requirements for

evaluation are identified, other resources such

as traceability and data collection tools have not

yet been identified.

Observation

Ability 3 Individuals performing evaluation activities have

experience or receive training.

Individuals performing evaluation have

experience in evaluation.

Strength

Ability 4 Members of the project team and groups

supporting the software acquisition receive

orientation on the objectives of the evaluation

approach.

Members of the project team receive orientation

on the objectives of the evaluation.

Strength

Activity 1 The project team performs its activities in

accordance with its documented evaluation

plans.

There is no documented evaluation plan. Weakness

Activity 2 The project’s evaluation requirements are

developed in conjunction with the development

of the system or software technical

requirements.

There is no evidence that evaluation

requirements were developed in conjunction

with system requirements.

Weakness

Activity 3 The evaluation requirements are incorporated

into the solicitation package and resulting

contract.

Requirements for the contractor to support the

evaluation are in the solicitation and contract.

Strength

Activity 4 The project team assesses the contractor’s

performance for compliance with evaluation

requirements.

The project team did not assess the contractor’s

performance for compliance with evaluation

requirements.

Weakness

(continued)

Page 18 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Feature Key practice Finding Rating

Activity 5 Planned evaluations are performed on the

acquired software products and services prior

to acceptance for operational use.

Since no documented evaluation plan exists,

there is no plan to follow.

Weakness

Activity 6 Results of the evaluations are analyzed and

compared to the contract’s requirements to

establish an objective basis to support the

decision to accept the products and services or

to take further action.

The FMS project has not yet reached the stage

where evaluation activities are analyzed and

compared to the contract.

Observation

Measurement 1 Measurements are made and used to determine

the status of the evaluation activities and

resultant products.

No measurements are made to determine the

status of evaluation activities.

Weakness

Verification 1 Evaluation activities are reviewed by acquisition

organization management on a periodic basis.

The Authority does not review the status of

evaluation activities.

Weakness

Verification 2 Evaluation activities are reviewed by the project

manager on both a periodic and event-driven

basis.

The project manager does not review the status

of evaluation activities.

Weakness

Acquisition Risk

Management

SEI defines risk as the possibility of suffering a loss. The purpose of

acquisition risk management is to formally identify risks as early as

possible and adjust the acquisition to mitigate those risks. An effective risk

management process, among other things, includes (1) having a written

policy on acquisition risk management, (2) developing a software

acquisition risk management plan, (3) conducting software risk

management activities in accordance with the plan (e.g., identifying risks,

taking mitigation actions, and tracking risk mitigation actions to

completion), and (4) measuring and reporting on the status of acquisition

risk management activities to management.

FMS Risk Management

Processes Are Ineffective

The FMS project had one strength for this KPA. The project has designated

responsibility for risk management to the project management team.

However, the District is not performing any of the other practices to

satisfy this KPA. For example, there is no written policy or plan for

acquisition risk management, resource requirements for risk management

have not been identified, and at the time of this audit, neither the Authority

nor the project manager were reviewing the activities for risk

management. Table 8 shows the strengths, weaknesses, and observations

for the acquisition risk management KPA and the specific findings

supporting these ratings.

Page 19 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Table 8: Acquisition Risk Management Findings

Feature Key practice Finding Rating

Commitment 1 The acquisition organization has a written policy

for the management of software acquisition risk.

There is no written policy for software

acquisition risk management.

Weakness

Commitment 2 Responsibility for software acquisition risk

management activities is designated.

Responsibility for software acquisition risk

management was designated to the acquisition

management contractor and the program office.

Strength

Ability 1 A group that is responsible for coordinating

software acquisition risk management activities

exists.

No group is responsible for coordinating

software acquisition risk management activities.

Weakness

Ability 2 Adequate resources are provided for software

acquisition risk management activities.

Resource requirements for acquisition risk

management have not yet been defined.

Weakness

Ability 3 Individuals performing software acquisition risk

management activities have experience or

receive required training.

Individuals designated to perform software

acquisition risk management do not have

experience and have not yet received training.

Weakness

Activity 1 Software acquisition risk management activities

are integrated into software acquisition planning.

Software acquisition risk management activities

were not integrated into software acquisition

planning.

Weakness

Activity 2 The software acquisition risk management plan

is developed in accordance with the project’s

defined software acquisition process.

A software acquisition risk management plan

has not been developed in accordance with a

defined software acquisition process.

Weakness

Activity 3 The project team performs its software

acquisition risk management activities in

accordance with its documented plans.

There is no documented acquisition risk

management plan.

Weakness

Activity 4 Risk management is conducted as an integral

part of the solicitation, project performance

management, and contract performance

management processes.

Risk management is not conducted as an

integral part of the solicitation, project

performance management, and contract

performance management processes.

Weakness

Activity 5 Software acquisition risk handling actions are

tracked and controlled until the risks are

mitigated.

Software acquisition risk handling actions are

not tracked and controlled until the risks are

mitigated.

Weakness

Measurement 1 Measurements are made and used to determine

the status of the acquisition risk management

activities and resultant products.

Measurements are not made for risk

management.

Weakness

Verification 1 Acquisition risk management activities are

reviewed by acquisition organization

management on a periodic basis.

Risk management activities are not reviewed by

the Authority.

Weakness

Verification 2 Acquisition risk management activities are

reviewed by the project manager on both a

periodic and event-driven basis.

Risk management activities are not reviewed by

the project manager.

Weakness

Conclusions Leading software acquisition organizations rely on defined and disciplined

software acquisition processes to deliver promised software capabilities

on time and within budget, first on a project-by-project basis, and later, as

the organization’s processes become more mature, consistently across the

Page 20 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

institution. While the District has many strengths in its acquisition

processes for FMS, it also has many weaknesses that, overall, make its

processes undisciplined and immature. As a result, the District’s success

or failure in acquiring FMS depends largely on specific individuals rather

than on well-defined software acquisition management practices. This

greatly reduces the probability that the system will consistently perform as

intended and be delivered on schedule and within budget.

To satisfy the intent of all the software acquisition key process areas and

thereby have a reasonable assurance that acquisition efforts are effectively

planned, managed, evaluated, and tracked, the District must address the

many weaknesses identified in this report. This would entail the District

formulating and implementing a written policy for software acquisition

planning, requirements development and management, project

management, contract tracking and oversight, evaluation, and acquisition

risk management. In addition, it is important for the District to track the

various activities for each KPA to ensure that they are being performed and

that evaluation and risk management activities are being planned and

effectively conducted.

Recommendations We recommend that the Chairman of the District of Columbia Financial

Responsibility and Management Assistance Authority direct the District’s

Chief Financial Officer to (1) take the following actions for the six KPAs we

reviewed to ensure that the current FMS acquisition and implementation is

satisfactorily completed and (2) apply these actions to any future software

acquisitions.

Software Acquisition Planning:

• Document decisions and update the planning documents to ensure that

large acquisitions such as FMS can be effectively managed.

• Designate responsibility for software acquisition planning activities.

• Determine required resources for acquisition planning.

• Ensure that measurements of software acquisition activities are taken.

• Ensure that the software acquisition planning documentation is updated as

well as make program changes regarding outsourcing of the data center

and upgrading the current system versus buying off-the-shelf.

• Ensure that the software acquisition planning documentation addresses

life-cycle support of the software.

• Develop a written policy for software acquisition planning.

Page 21 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

Requirements Development and Management:

• Develop an organizational policy for establishing and managing

software-related requirements.

• Clearly assign responsibility for requirements development and

management.

• Document either resource requirements or resources expended for

requirements development activities.

• Develop the capability to trace between contractual requirements and the

contractor’s work products.

• Develop measurements to determine the status of the requirements

development and management activities.

Project Management:

• Develop a written policy for the execution of the software project.

• Authorize the project manager to independently alter either the

performance, cost, or schedule.

• Require that measurements be taken to determine the status of project

management activities.

Contract Tracking and Oversight:

• Develop written policy for contract tracking and oversight activities for

the financial management system project.

• Support the project team with contracting specialists.

• Require that the project team review the contractor’s planning documents

(for example, the project management plan, software risk management

plan, software engineering plan, configuration management plan).

• Assign someone responsibility for maintaining the integrity of the contract.

• Take measurements to determine the status of contract tracking and

oversight activities.

Evaluation:

• Develop written policy for managing the evaluation of acquired software

products and services.

• Develop a documented evaluation plan.

• Develop evaluation requirements in conjunction with system

requirements.

Page 22 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

• Assess the contractor’s performance for compliance with evaluation

requirements.

• Develop measurements to determine the status of evaluation activities.

• Ensure that the Authority and the project manager review the status of

evaluation activities.

Acquisition Risk Management:

• Develop written policy for software acquisition risk management.

• Designate a group to be responsible for coordinating software acquisition

risk management activities.

• Define resource requirements for acquisition risk management.

• Ensure that individuals designated to perform software acquisition risk

management have adequate experience and training.

• Integrate software acquisition risk management activities into software

acquisition planning.

• Develop a software acquisition risk management plan in accordance with

a defined software acquisition process.

• Develop a documented acquisition risk management plan and conduct risk

management as an integral part of the solicitation, project performance

management, and contract performance management processes.

• Track and control software acquisition risk handling actions until the risks

are mitigated.

• Ensure that risk management activities are reviewed by the Authority and

the project manager.

Agency Comments

and Our Evaluation

GAO requested comments on a draft of this report from the Chairman,

District of Columbia Financial Responsibility and Management Assistance

Authority, and the District’s Chief Financial Officer. They provided us with

written comments that are reprinted in appendixes I and II.

In their comments, the District of Columbia Financial Responsibility and

Management Assistance Authority’s Executive Director and the District of

Columbia’s Chief Financial Officer acknowledged that the software

acquisition project for the new financial management system was a high

risk initiative and that the District’s processes were not sufficiently

mature. The District Chief Financial Officer identified initiatives in each of

the key process areas. Both cited ongoing corrective actions, which, if

properly implemented, will address several of our recommendations. For

example, the Chief Financial Officer stated that the District is developing a

Risk Management Plan and is evaluating various strategies to identify and

Page 23 GAO/AIMD-98-88 D.C. Financial Management System
B-279266 

manage risks, and that the Chief Technology Officer for the District of

Columbia is developing policies and procedures for information resource

management which will include software acquisition.

However, the District Chief Financial Officer also added that their efforts

to date have achieved a sound acquisition state consistent with the intent

of the SA-CMM. As discussed in the report, significant improvements would

be necessary to achieve the minimally acceptable level of maturity as

defined by the Software Engineering Institute’s Software Acquisition

Maturity Model to satisfy the intent of all the software acquisition key

process areas. Accordingly, the District has not yet achieved a sound

acquisition state consistent with the intent of the SA-CMM. If the District is

to instill the needed discipline into its systems acquisition processes

consistent with the intent of SA-CMM, it will need to effectively implement

all of our recommendations.

We are sending copies of this report to the Ranking Minority Member of

your Subcommittee and to the Chairmen and Ranking Minority Members

of the Subcommittee on Oversight of Government Management,

Restructuring, and the District of Columbia, Senate Committee on

Governmental Affairs, the Subcommittee on the District of Columbia,

Senate Committee on Appropriations, and the Subcommittee on the

District of Columbia, House Committee on Government Reform and

Oversight. We are also sending copies to the Director of the Office of

Management and Budget, the Chairman of the District of Columbia

Financial Responsibility and Management Assistance Authority, and the

Chief Financial Officer of the District of Columbia. Copies will be made

available to others upon request. If you have questions or wish to discuss

the issues in this report, please contact me at (202) 512-6412. Major

contributors to this report are listed in appendix III.

Sincerely yours,

Dr. Rona B. Stillman

Chief Scientist for

Computers and Telecommunications

Page 24 GAO/AIMD-98-88 D.C. Financial Management System
Page 25 GAO/AIMD-98-88 D.C. Financial Management System
Contents

Letter 1

Appendix I 

Comments From the

Chairman, District of

Columbia Financial

Responsibility and

Management

Assistance Authority

28

Appendix II 

Comments From the

District of Columbia

Chief Financial

Officer

30

Appendix III 

Major Contributors to

This Report

35

Tables Table 1: SA-CMM KPAs Used to Assess the FMS Project Software

Acquisition Maturity

4

Table 2: Software Acquisition Planning Findings 9

Table 3: Solicitation Findings 10

Table 4: Requirements Development and Management Findings 13

Table 5: Project Management Findings 15

Table 6: Contract Tracking and Oversight Findings 16

Table 7: Evaluation Findings 18

Table 8: Acquisition Risk Management Findings 20

Figures Figure 1: SA-CMM Levels and Descriptions 3

Figure 2: Key Process Area Results 7

Page 26 GAO/AIMD-98-88 D.C. Financial Management System
Contents

Abbreviations

FMS financial management system

KPA key process area

SA-CMM Software Acquisition Capability Maturity Model

SCE Software Capability Evaluation

SEI Software Engineering Institute

Page 27 GAO/AIMD-98-88 D.C. Financial Management System
Appendix I 

Comments From the Chairman, District of

Columbia Financial Responsibility and

Management Assistance Authority

Page 28 GAO/AIMD-98-88 D.C. Financial Management System
Appendix I 

Comments From the Chairman, District of

Columbia Financial Responsibility and

Management Assistance Authority

Page 29 GAO/AIMD-98-88 D.C. Financial Management System
Appendix II 

Comments From the District of Columbia

Chief Financial Officer

Page 30 GAO/AIMD-98-88 D.C. Financial Management System
Appendix II 

Comments From the District of Columbia

Chief Financial Officer

Page 31 GAO/AIMD-98-88 D.C. Financial Management System
Appendix II 

Comments From the District of Columbia

Chief Financial Officer

Page 32 GAO/AIMD-98-88 D.C. Financial Management System
Appendix II 

Comments From the District of Columbia

Chief Financial Officer

Page 33 GAO/AIMD-98-88 D.C. Financial Management System
Appendix II 

Comments From the District of Columbia

Chief Financial Officer

Page 34 GAO/AIMD-98-88 D.C. Financial Management System
Appendix III 

Major Contributors to This Report

Accounting and

Information

Management Division,

Washington, D.C.

Keith Rhodes, Technical Director

Nabajyoti Barkakati, Assistant Director

Hodge Herry, Assistant Director

John C. Martin, Assistant Director

Prithviraj Mukherji, Assistant Director

Madhav Panwar, Assistant Director

Norma Samuel, Project Manager

Office of General

Counsel

Richard Cambosos, Senior Attorney

(916247) Page 35 GAO/AIMD-98-88 D.C. Financial Management System
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100