Report to the Ranking Minority

Member, Subcommittee on Housing,

and Transportation, Committee on

Banking, Housing and Urban Affairs,

U.S. Senate

United States General Accounting Office

GAO

September 2001 HUD INFORMATION

SYSTEMS

Immature Software

Acquisition Capability

Increases Project

Risks

GAO-01-962
Page i GAO-01-962 HUD Information Systems

Letter 1

Executive Summary 2

Chapter 1 Introduction 7

HUD Relies on Information Systems 7

The Software Acquisition Capability Maturity Model Provides a

Means of Assessing an Organization’s Ability to Manage

Software Acquisitions 9

Objective, Scope, and Methodology 12

Chapter 2 HUD’s Requirements Development and Management

Processes Are Not Repeatable 15

Chapter 3 HUD’s Project Management Processes Are Not

Repeatable 28

Chapter 4 HUD’s Contract Tracking and Oversight Processes

Are Not Repeatable 41

Chapter 5 HUD’s Software Evaluation Processes Are Not

Repeatable 54

Chapter 6 Conclusions, Recommendations, and Agency

Comments 67

Appendix I Comments From the Department of Housing and

Urban Development 69

Contents
Page ii GAO-01-962 HUD Information Systems

Appendix II GAO Contact and Staff Acknowledgments 71

Tables

Table 1: Total Number of Key Process Area Strengths, Weaknesses,

and Observations for the Five Projects 5

Table 2: Key Process Areas for SA-CMM Level 2 11

Table 3: Requirements Development and Management Findings for

the Public and Indian Housing Information Center System 18

Table 4: Requirements Development and Management Findings for

the Real Estate Management System 20

Table 5: Requirements Development and Management Findings for

the Resident Assessment Subsystem 22

Table 6: Requirements Development and Management Findings for

HUD’s Central Accounting and Program System 24

Table 7: Requirements Development and Management Findings for

the Empowerment Information System 26

Table 8: Project Management Findings for the Public and Indian

Housing Information Center System 31

Table 9: Project Management Findings for the Real Estate

Management System 33

Table 10: Project Management Findings for the Resident

Assessment Subsystem 35

Table 11: Project Management Findings for HUD’s Central

Accounting and Program System 37

Table 12: Project Management Findings for the Empowerment

Information System 39

Table 13: Contract Tracking and Oversight Findings for the Public

and Indian Housing Information Center System 44

Table 14: Contract Tracking and Oversight Findings for the Real

Estate Management System 46

Table 15: Contract Tracking and Oversight Findings for the

Resident Assessment Subsystem 48

Table 16: Contract Tracking and Oversight Findings for HUD’s

Central Accounting and Program System 50

Table 17: Contract Tracking and Oversight Findings for the

Empowerment Information System 52

Table 18: Software Evaluation Findings for the Public and Indian

Housing Information Center System 57

Table 19: Software Evaluation Findings for the Real Estate

Management System 59
Page iii GAO-01-962 HUD Information Systems

Table 20: Software Evaluation Findings for the Resident

Assessment Subsystem 61

Table 21: Software Evaluation Findings for HUD’s Central

Accounting and Program System 63

Table 22: Software Evaluation Findings for the Empowerment

Information System 65

Figures

Figure 1: SA-CMM Levels and Descriptions 10

Figure 2: Requirements Development and Management Summary 17

Figure 3: Project Management Summary 30

Figure 4: Contract Tracking and Oversight Summary 43

Figure 5: Software Evaluation Summary 56

Abbreviations

CIO Chief Information Officer

EIS Empowerment Information System

GAO General Accounting Office

HUD Department of Housing and Urban Development

HUDCAPS HUD Central Accounting and Program System

PIC Public and Indian Housing Information Center system

RASS Resident Assessment Subsystem

REMS Real Estate Management System

SA-CMM Software Acquisition Capability Maturity Model

SEI Software Engineering Institute
Page 1 GAO-01-962 HUD Information Systems

September 14, 2001

The Honorable Wayne Allard

Ranking Minority Member

Subcommittee on Housing and Transportation

Committee on Banking, Housing, and Urban Affairs

United States Senate

Dear Senator Allard:

The accompanying report is the first in a series of reports responding to

your February 2001 request that we examine a range of management

issues at the Department of Housing and Urban Development. This report

discusses our assessment of the department’s capability to acquire

software. We are making recommendations to the Secretary of Housing

and Urban Development to assist the department in improving this

capability.

We are sending copies of this report to the Secretary of Housing and

Urban Development, the Director of the Office of Management and

Budget, and other congressional committees. We will also make copies

available to other interested parties upon request.

If you have questions or wish to discuss the issues in this report, please

contact me at (202) 512-6240 or by email at koontzl@gao.gov. An

additional GAO contact and staff acknowledgements are listed in

appendix II.

Sincerely yours,

Linda D. Koontz

Director, Information Management Issues

United States General Accounting Office

Washington, DC 20548
Executive Summary

Page 2 GAO-01-962 HUD Information Systems

The Department of Housing and Urban Development (HUD) routinely

acquires new information systems and enhancements to manage and

support its various programs and operations. GAO has designated HUD’s

major program areas as high risk, in part because the department’s

information and financial management systems were poorly integrated,

ineffective, and generally unreliable.1

 HUD has been endeavoring to

improve its systems to better support its missions and management

reforms.

Because of the importance of information and financial management

systems and related improvement efforts to meeting HUD’s mission, the

Ranking Minority Member, Subcommittee on Housing and Transportation,

Senate Committee on Banking, Housing, and Urban Affairs, requested that

GAO review the maturity of HUD’s software acquisition processes. More

mature processes increase the likelihood that the software acquired will

meet an organization’s needs.

HUD is the principal federal agency responsible for housing and

community development programs. HUD’s mission includes making

housing affordable (through mortgage insurance for multifamily housing

and through rental assistance), helping to revitalize localities, and

encouraging home ownership. In fiscal year 2001, the department’s budget

was about $32 billion, including $360 million for information technology

investments.

High-quality software is essential for HUD’s information systems to

provide reliable management, financial, and administrative information

and support the department’s many programs. The quality of software is

governed largely by the quality of the processes involved in developing or

acquiring it and in maintaining it. Carnegie Mellon University’s Software

Engineering Institute (SEI), recognized for its expertise in software

processes, has developed models and methods that define and determine

organizations’ software process maturity. Together, these models and

 1

GAO High-Risk Program (GAO/AIMD-94-72R, January 27, 1994); High-Risk Series:

Department of Housing and Urban Development (GAO/HR-95-11, February 1995); HighRisk Series: Department of Housing and Urban Development (GAO/HR-97-12, February

1997); Major Management Challenges and Program Risks: Department of Housing and

Urban Development (GAO/OCG-99-8, January 1999); Major Management Challenges and

Program Risks: Department of Housing and Urban Development (GAO-01-248, January

2001).

Executive Summary

Purpose

Background
Executive Summary

Page 3 GAO-01-962 HUD Information Systems

methods provide a logical framework for baselining an organization’s

current process capabilities (i.e., determining what practices are

effectively implemented (strengths), not effectively implemented

(weaknesses), or contain mixed or inconclusive evidence (observations))

and providing a structured plan for incremental process improvement.

Using SEI’s software acquisition capability maturity modelSM and SEI’s

software capability evaluation method,2

 GAO analysts trained at SEI

evaluated HUD’s software acquisition maturity in four of seven key

process areas3

 that are necessary to attain a “repeatable” level of process

maturity. The “repeatable” level of process maturity is the second level on

SEI’s five-level scale. An organization at the repeatable level of process

maturity has the necessary process discipline in place to repeat earlier

successes on similar projects. Organizations that do not satisfy the

requirements for the repeatable level are by default judged to be at the

“initial” level of maturity. This means that their processes are immature, ad

hoc, and sometimes even chaotic, with few of the processes defined and

success dependent mainly on the heroic efforts of individuals.

In this evaluation, GAO examined five ongoing projects selected by HUD

as being representative of the department’s current software acquisition

practices.4

 These were the Public and Indian Housing Information Center

system, the Real Estate Management System, the Resident Assessment

Subsystem, HUD’s Central Accounting and Program System, and the

Empowerment Information System.

HUD did not fully satisfy the requirements for any of the “repeatable” key

process areas we reviewed. While HUD has several software acquisition

 2

 Capability Maturity ModelSM is the service mark of Carnegie Mellon University, and CMM

is registered in the U.S. Patent and Trademark Office. GAO used Software Acquisition

Capability Maturity Model (SA-CMM) Version 1.2 (CMU/SEI-99-TR-002, April 1999), the

latest version of the model.

3

 The four key process areas GAO evaluated are requirements development and

management, project management, contract tracking and oversight, and software

evaluation. GAO did not evaluate the remaining three key process areas—software

acquisition planning, solicitation, and transition to support—because HUD’s project teams

had not recently performed activities in these areas.

4

 GAO also asked that the five projects be (1) major efforts with large software acquisition

components, (2) managed by integrated project teams, (3) at different stages of the life

cycle, and (4) among HUD’s best managed projects.

Results in Brief
Executive Summary

Page 4 GAO-01-962 HUD Information Systems

process strengths, GAO found a large number of software process

weaknesses in all key process areas evaluated: requirements development

and management, project management, contract tracking and oversight,

and software evaluation. As a result, its processes for acquiring software

are immature and can be characterized as ad hoc, sometimes chaotic, and

not repeatable across projects. These weaknesses can lead to systems that

do not meet the information needs of management and staff, do not

provide support for needed programs and operations, and cost more and

take longer than expected to complete. HUD is aware that it has software

acquisition weaknesses, has stated its commitment to improving its

software and system acquisition processes, and will begin a process

improvement effort in the near future.

HUD did not meet the requirements for the repeatable level of maturity in

the four key process areas we reviewed. Of the 310 key practices GAO

evaluated, 50 percent constituted strengths (i.e., key practice was

effectively implemented), 39 percent were weaknesses (i.e., key practice

was not effectively implemented), 10 percent were rated as observations,

and 1 percent were not rated.

Certain weaknesses were systemic, recurring in most or all of the key

process areas. For example, HUD has no overall policy for the acquisition

of software; the majority of project teams did not develop plans to conduct

various activities; none of the teams measured the status of activities; the

majority of the teams did not report regularly to management on progress

and problems; and most project managers did not require regular reports

on progress and problems. These weaknesses can lead to systems that do

not meet HUD’s information needs, do not effectively support HUD’s

programs and services, and cost more and take longer to complete.

To reach the repeatable level of maturity, HUD must overcome the key

practice weaknesses identified in this report. Table 1 summarizes the

strengths and weaknesses for the five projects.

Principal Finding:

HUD’s Software

Acquisition Processes

Are Immature
Executive Summary

Page 5 GAO-01-962 HUD Information Systems

Table 1: Total Number of Key Process Area Strengths, Weaknesses, and

Observations for the Five Projects

Key practice ratings

Key process area Strengths Weaknesses Observations Not rated

Requirements

development and

management 27 31 12 0

Project management 45 25 10 0

Contract tracking and

oversight 45 33 4 3

Evaluation 37 33 5 0

Total 154 122 31 3

HUD acknowledged that it has software acquisition weaknesses and stated

its commitment to improving its software and system acquisition

processes. The department has prepared a statement of work to obtain

contractor support to begin a software process improvement effort. One of

the tasks the contract will include is development of a software process

improvement plan.

To successfully complete such an effort, HUD’s plan must address several

important issues. To be comprehensive, this plan should include the

results of this review, measurable goals and time frames, estimates of

resource requirements, and time frames to reach the repeatable level. In

addition, it should address the systemic weaknesses that GAO found.

Finally, the plan should include steps to address the key process areas

GAO did not review, as well as steps to ensure that all ongoing and new

software acquisition projects adopt processes that meet the requirements

for the repeatable level.

To improve HUD’s software acquisition capabilities, GAO recommends

that the Secretary of Housing and Urban Development direct the HUD

Chief Information Officer to develop and implement a comprehensive plan

for software acquisition process improvement that is based on the

software capability results in this report and specifies measurable goals

and time frames, sets priorities for initiatives, estimates resource

requirements (for trained staff and funding), and defines a process

improvement management structure.

Also, to address the systemic weaknesses mentioned above, the plan

should contain steps to

Recommendations for

Executive Action
Executive Summary

Page 6 GAO-01-962 HUD Information Systems

• develop a comprehensive policy for the acquisition of software,

• require plans for specific acquisition activities,

• measure and track the status of activities performed by the software

acquisition teams, and

• report regularly to management on progress and problems.

In addition, to ensure that all aspects of software acquisition are

addressed, we recommend that the Secretary direct the CIO to

• assess HUD’s maturity in the three key process areas that could not be

evaluated by GAO and include any needed improvement actions in the

comprehensive plan for software acquisition process improvement;

• ensure that all new software acquisition projects in HUD adopt processes

that satisfy at least SA-CMM level 2 requirements; and

• ensure that process improvement activities are initiated for all ongoing

software acquisition projects.

In providing written comments on a draft of our report, HUD agreed with

our assessment of its software acquisition processes and our

recommendations for strengthening them, and stated that our assessment

and recommendations will be addressed as part of its process

improvement effort. This effort should help the department improve its

ability to acquire systems that meet management and staff information

needs and efficiently support programs and operations.

Agency Comments
Chapter 1: Introduction

Page 7 GAO-01-962 HUD Information Systems

The Department of Housing and Urban Development (HUD), established

in 1965, has as its primary mission ensuring a decent, safe, and sanitary

home and suitable living environment for every American. HUD’s mission

includes making housing affordable for about 4 million low-income

households by insuring loans for multifamily rental housing and by

providing rental assistance. Its mission also includes helping to revitalize

over 4,000 localities through community development programs. The

department encourages homeownership by providing, through its Federal

Housing Administration, mortgage insurance for about 7 million

homeowners who otherwise might not have qualified for loans. HUD is

one of the nation’s largest financial institutions, responsible for managing

about $508 billion in insured mortgages and $570 billion in guarantees of

mortgage-backed securities. HUD’s budget in fiscal year 2001 was about

$32 billion, including $360 million for information technology investments.

Like many federal agencies, HUD relies on its information and financial

management systems to help carry out its missions. However, past reports

by us and others showed that HUD was plagued by poorly integrated,

ineffective, and generally unreliable information systems that did not

satisfy management needs or provide adequate controls.1

 We designated

HUD’s program areas as high risk in part because of widespread problems

with information and financial management systems. In 1997, HUD began

a management reform effort designed to transform its culture, manage its

programs and staff more effectively, ultimately restore public trust in the

department, and modernize it for the 21st century.

In January 2001, we reported that HUD had been making credible progress

towards addressing its management reform goals.2

 However, we also

reported that information and financial systems remained an area of

management challenge because of unresolved issues. In addition, in its

March 1, 2001, financial statement audit report, the Office of the HUD

Inspector General listed two material internal control weaknesses related

to systems. The report stated that HUD needs to (1) complete

improvements to its financial management systems to meet its program

and financial management needs and (2) enhance the Federal Housing

 1 GAO-01-248, January 2001; GAO/OCG-99-8, January 1999; GAO/HR-97-12, February 1997;

GAO/HR-95-11, February 1995; GAO/AIMD-94-72R, January 27, 1994.

2 GAO-01-248, January 2001.

Chapter 1: Introduction

HUD Relies on

Information Systems
Chapter 1: Introduction

Page 8 GAO-01-962 HUD Information Systems

Administration’s information systems to more effectively support its

business practices.

HUD is aware that information systems remain an area of challenge. To

help address some of the challenges, HUD plans to (1) institute more

disciplined processes in its information technology capital investment

planning reviews (quarterly reviews that assess how well projects are

functioning and attempt to alert management to potential problems),

(2) continue training its project managers in new project management

techniques, (3) complete efforts to bring all software products under

standard change control and configuration management3

 and designate a

separate organization to address configuration management issues,

(4) improve HUD’s end-to-end testing4

 capabilities, and (5) obtain and

deploy new software for project-level status reporting.

To manage specific software acquisitions, HUD uses integrated project

teams. These teams include representatives from the business

organization(s) that will use the acquired software to do their work;

information technology staff to oversee the contractor and review

contractor work products; and contract and procurement staff to assist

the team in contract management. Policy and procedures for software

acquisition are the responsibility of HUD’s Chief Information Officer.

 3

 Configuration management is a discipline applying technical and administrative direction

and surveillance to identify and document the functional and physical characteristics of

hardware or software, control changes to these characteristics, record and report change

processing, and verify compliance of hardware or software with specified requirements.

4

 End-to-end testing is done to verify that a defined set of interrelated systems, which

collectively support an organizational core business area or function, interoperate as

intended in an operational environment. This includes not only those systems owned and

managed by the organization but also those external systems with which they interface.
Chapter 1: Introduction

Page 9 GAO-01-962 HUD Information Systems

The software acquisition capability maturity model (SA-CMM),5 developed

by Carnegie Mellon’s Software Engineering Institute (SEI), is used to

measure an organization’s capability to manage the acquisition of

software. SEI’s expertise in and methods for software process assessment

are recognized and accepted worldwide throughout the industry. The

model defines five levels of software acquisition maturity. Each level of

maturity (except level 1) indicates process capability and identifies key

process areas. For a maturity level to be achieved, all key process areas

related to that level must be implemented effectively.

The first level of process capability is level 2 (referred to as the repeatable

level), where basic management processes are established to track

performance, cost, and schedule, and the necessary discipline is in place

to repeat earlier successes on similar projects. Organizations that do not

effectively implement all key process areas for the repeatable level are, by

default, at level 1 or the initial level of maturity. Level 1 processes can be

described as immature, ad hoc, and sometimes chaotic; success in

software acquisition for these organizations is usually dependent upon the

ability and commitment of the staff involved. Figure 1 explains further the

five-level software acquisition model.

 5 Software Acquisition Capability Maturity Model (SA-CMM), Version 1.2, Software

Engineering Institute, CMU/SEI-99-TR-002 (April 1999).

The Software

Acquisition Capability

Maturity Model

Provides a Means of

Assessing an

Organization’s Ability

to Manage Software

Acquisitions
Chapter 1: Introduction

Page 10 GAO-01-962 HUD Information Systems

Figure 1: SA-CMM Levels and Descriptions

Table 2 identifies each of the seven key process areas associated with the

repeatable level of maturity.

The software acquisition process is characterized as

ad hoc, and occasionally even chaotic. Few processes

are defined and success depends on individual effort.

Level 1: Initial

Basic project management processes are established

to track performance, cost, and schedule. The

necessary process discipline is in place to repeat

earlier successes on projects in similar domains.

Level 2: Repeatable

The acquisition organization’s software acquisition process

is documented, standardized, and established as the

standard software acquisition process. All projects use an

approved, tailored version of the organization’s standard

software acquisition process for acquiring their software

products and services.

Level 3: Defined

Level 5: Optimizing

Level 4: Managed

Continuous process improvement is empowered by

quantitative feedback from the process and from piloting

innovative ideas and technologies.

Detailed measures of quality of the software acquisition

processes, products, and services are collected. The

software processes, products, and services are

quantitatively understood and controlled.

Disciplined

process

Standard

consistent

process

Predictable

process

Continuously

improving

process
Chapter 1: Introduction

Page 11 GAO-01-962 HUD Information Systems

Table 2: Key Process Areas for SA-CMM Level 2

SA-CMM level 2 key

process areas Description

Software acquisition

planning

Ensuring that reasonable planning for the software

acquisition is conducted and that all elements of the

project are included.

Solicitation Ensuring that award is made to the contractor most

capable of satisfying the specified requirements.

Requirements development

and management

Establishing a common and unambiguous definition of

software acquisition requirements that is understood by

the acquisition team, system users, and contractor(s).

Project management Managing the activities of the project office and

supporting contractor(s) to ensure a timely, efficient, and

effective software acquisition.

Contract tracking and

oversight

Ensuring that the software activities under contract are

being performed in accordance with contract

requirements and that products and services will satisfy

contract requirements.

Evaluation Determining that the acquired software products and

services satisfy contract requirements before acceptance.

Transition to support Providing for the transition of the software products being

acquired to their eventual support organization.

As established by the model, each key process area contains five common

features—commitment to perform, ability to perform, activities

performed, measurement and analysis, and verifying implementation—that

indicate whether the implementation and institutionalization of the key

process area can be effective, repeatable, and lasting. The common feature

definitions are as follows:

• Commitment to perform: This feature describes the actions that the

organization takes to establish the process and ensure that it can endure.

Key practices typically involve establishing organizational policies and

sponsorship.

• Ability to perform: This feature describes the preconditions that must

exist in the project or organization to competently implement the software

acquisition process. Key practices typically include assigning

responsibility and providing training.

• Activities performed: This feature describes the roles and procedures

necessary to implement a key process area. Key practices typically involve

establishing plans and procedures, performing the work, tracking it, and

taking appropriate management actions.

• Measurement and analysis: This feature describes the activities

necessary to measure progress and analyze the measurements. Key

practices typically involve defining the measurements to be taken and the
Chapter 1: Introduction

Page 12 GAO-01-962 HUD Information Systems

analyses to be conducted to determine the status and effectiveness of the

activities performed.

• Verifying implementation: This feature describes the steps the

organization must take to ensure that project activities are performed in

accordance with established processes. Key practices typically involve

regular reviews by management.

Each common feature consists of a number of key practices—specific

activities such as developing an organizational policy for software

acquisition, developing various plans for software acquisition activities,

and tracking a contractor’s progress. When an organization is evaluated

against the SA-CMM, comparisons of actual performance against a key

practice can result in one of four possible outcomes or ratings:

• Strength: The key practice involved was effectively implemented.

• Weakness: The key practice was ineffectively implemented or was not

implemented.

• Observation: The key practice was evaluated, but cannot be characterized

as a strength because the (1) project team did not provide sufficient

evidence to support a strength rating or (2) key practice was only partially

performed.

• Not rated: The key practice is not relevant to the project and was therefore

not rated.

To achieve the repeatable level, HUD would have to demonstrate that the

key practices related to that level were implemented effectively in its

software acquisition projects.

The Ranking Minority Member of the Subcommittee on Housing and

Transportation, Senate Committee on Banking, Housing, and Urban

Affairs, requested that we review HUD’s software acquisition capabilities.

Our objective for this review was to determine the maturity of HUD’s

software acquisition processes.

To determine HUD’s software acquisition process maturity, we applied the

Software Engineering Institute’s SA-CMM using our SEI-trained analysts.

We focused on the key process areas necessary to obtain a repeatable

level of maturity, the second level of SEI’s five-level model. We met with

project managers and project team members to determine whether and to

Objective, Scope, and

Methodology
Chapter 1: Introduction

Page 13 GAO-01-962 HUD Information Systems

what extent they implemented each key practice and to obtain relevant

documentation. In accordance with the SEI model, for each key process

area we reviewed,6 we evaluated HUD’s institutional policies and practices

and compared project-specific guidance and practices against the required

key practices.

More specifically, for each key practice we reviewed, we compared

project-specific documentation and practices against the criteria in the

software acquisition model. If the project met the criteria for the key

practice reviewed, we rated it as a strength. If the project did not meet the

criteria for the key practice reviewed, we rated it as a weakness. If the

evidence did not support a rating of a strength or a weakness, we rated it

as an observation. If the key practice was not relevant to a project, we

stated that the key practice was not rated.

We evaluated five HUD projects, each of which is described below. We

asked HUD to select five system acquisition projects that were

representative of HUD software acquisition practices and met the

following criteria: all were (1) major efforts with large software acquisition

components, (2) managed by integrated project teams, (3) at different

stages of the life cycle, and (4) among the best-managed acquisitions. HUD

selected the following projects:

• Public and Indian Housing Information Center system (PIC)—an Internetbased system that collects information about the public housing inventory

managed by HUD and automates the processes for allocating program

funds to public housing authorities. It is a central repository for

information about public and Indian housing events and program areas as

well as for the housing inventory.

• Real Estate Management System (REMS)—a system that provides

automated support to collect and maintain data on multifamily insured

and assisted properties and their ownership/management entities and

provides access to data collected by HUD on physical and financial

assessments.

 6

We evaluated HUD on four of the seven key process areas—requirements development

and management, project management, contract tracking and oversight, and software

evaluation. We did not evaluate HUD on the three other key process areas—software

acquisition planning, solicitation, and transition to support—because these were not

recently performed for the selected projects.
Chapter 1: Introduction

Page 14 GAO-01-962 HUD Information Systems

• Resident Assessment Subsystem (RASS)—a system that collects

information on public housing from residents and manages the resident

assessment process.

• HUD Central Accounting and Program System (HUDCAPS)—HUD’s core

accounting and general ledger system. It serves as the standard accounting

system for all program areas.

• Empowerment Information System (EIS)—an enterprisewide data

warehouse project that is to provide HUD users and business partners

with access to reports and analytical information across a large variety of

program data sources.

We performed our work from March 2001 through August 2001 in

accordance with generally accepted government auditing standards. We

provided a draft of our report to HUD for review, and the department’s

comments are addressed in chapter 6 and reprinted as appendix I.
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 15 GAO-01-962 HUD Information Systems

HUD did not satisfy the criteria for the repeatable level of maturity in this

key process area because of the number and severity of key practice

weaknesses found. As a result of these weaknesses, HUD is at greater risk

of acquiring software products that do not meet mission needs and that

exceed cost, schedule, and performance goals.

The purpose of requirements development and management is to establish

and maintain a common and unambiguous definition of software

requirements1

 among the acquisition team, system users, and software

development contractor. Within this key process area, the SA-CMM

specifies key practices for each of the five common features that an

organization must implement effectively to achieve the repeatable level of

maturity. Generally, these practices include having a written

organizational policy for establishing and managing requirements

allocated to software, documenting plans for the development and

management of requirements, having documented processes for

requirements development, including elicitation, analysis, and verification,

measuring and reporting on the status of activities to management,

appraising the impact of system-level requirements changes on the

software, and having a mechanism to ensure that contractor-delivered

work products meet the specified requirements.

Of the 70 key practices for this key process area, HUD had 27 strengths, 31

weaknesses and 12 observations. For the

• Commitment to perform feature, there were two strengths, five

weaknesses, and three observations.

• Ability to perform feature, there were nine strengths, five weaknesses, and

one observation.

• Activities performed feature, there were ten strengths, 14 weaknesses, and

six observations.

• Measurement and analysis feature, there were one strength, three

weaknesses, and one observation.

• Verifying implementation feature, there were five strengths, four

weaknesses, and one observation.

In addition, none of the projects had strengths in all the key practices.

 1

 Requirements describe the functions that the software will perform to meet user needs

and provide support for business processes.

Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 16 GAO-01-962 HUD Information Systems

As a result of these weaknesses, HUD is exposed to increased risks that

acquired software will not meet its needs and that projects will exceed

cost, schedule, and performance goals. HUD acknowledges the need to

improve its requirements development and management processes and is

committed to improving its software and systems acquisition capability.

The department has prepared a statement of work to obtain contractor

support to begin a software process improvement effort. This document

specifies that the contractor would review HUD software acquisition

processes, prioritize the key processes and practices to address, and

develop a plan to improve HUD’s acquisition processes.

Details of our evaluation are provided in the following figure and tables.

Figure 2 provides a comprehensive listing of the strengths, weaknesses,

and observations for the requirements development and management key

process area. The specific findings supporting the practice ratings cited in

figure 2 are in tables 3 through 7.
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 17 GAO-01-962 HUD Information Systems

Figure 2: Requirements Development and Management Summary

Common Projects

features Key practices PIC REMS RASS HUDCAPS EIS

Commitment to

perform 1

The acquisition organization has a written policy for

establishing and managing the software-related

contractual requirements.

Weakness Weakness Observation Observation Weakness

Commitment to

perform 2

Responsibility has been designated for requirements

development and management activities. Weakness Strength Strength Observation Weakness

Ability to

perform 1

A group is responsible for performing requirements

development and management activities. Weakness Strength Strength Observation Strength

Ability to

perform 2

Adequate resources are provided for requirements

development and management activities. Weakness Weakness Strength Weakness Weakness

Ability to

perform 3

Individuals performing requirements development and

management activities have experience or receive

training.

Strength Strength Strength Strength Strength

Activity

performed 1

The project team performs its activities in accordance

with its documented requirements development and

management plans.

Weakness Observation Strength Weakness Weakness

Activity

performed 2

The project team develops, baselines, and maintains

software-related contractual requirements and places

them under change control early in the project, but not

later than the release of the solicitation package.

Observation Weakness Strength Strength Observation

Activity

performed 3

The project team appraises system requirements

change requests for their impact on the software being

acquired.

Weakness Weakness Strength Strength Weakness

Activity

performed 4

The project team appraises changes to the softwarerelated contractual requirements for their impact on

performance, architecture, supportability, system

resource utilization, and contract schedule and cost.

Weakness Weakness Strength Weakness Weakness

Activity

performed 5

Bidirectional traceability between the contractual

requirements and the contractor team’s software

products and services is maintained throughout the

effort.

Observation Weakness Strength Strength Weakness

Activity

performed 6

The end user and other affected groups are involved in

the development of all software-related contractual

requirements and any subsequent change activity.

Observation Strength Strength Observation Weakness

Measurement

and analysis 1

Measurements are made and used to determine the

status of the requirements development and

management activities and resultant products.

Weakness Weakness Observation Weakness Strength

Verifying

implementation 1

Requirements development and management activities

are reviewed by the acquisition organization on a

periodic basis.

Strength Observation Weakness Strength Strength

Verifying

implementation 2

Requirements development and management activities

are reviewed by the project manager on both a periodic

and an event-driven basis.

Weakness Strength Weakness Strength Weakness

Key Strength Strength: Key practice effectively implemented.

Weakness Weakness: Key practice not effectively implemented.

Observation Observation: Key practice evaluated; evidence not sufficient to rate as a strength, or practice only partially performed.
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 18 GAO-01-962 HUD Information Systems

Table 3: Requirements Development and Management Findings for the Public and Indian Housing Information Center System

Public and Indian Housing Information Center system

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a written

policy for establishing and managing the

software-related contractual

requirements.

HUD has no written policy for establishing

and managing software-related

contractual requirements.

Weakness

Commitment to perform 2 Responsibility has been designated for

requirements development and

management activities.

Responsibility for requirements

development and management has not

been designated.

Weakness

Ability to perform 1 A group is responsible for performing

requirements development and

management activities.

No group is assigned responsibility for

performing requirements development

and management.

Weakness

Ability to perform 2 Adequate resources are provided for

requirements development and

management activities.

There is no mechanism to identify

resource needs and ensure that they are

provided to the project. Team members

indicated that they do not have adequate

resources for performing these activities.

Weakness

Ability to perform 3 Individuals performing requirements

development and management activities

have experience or receive training.

Project team members performing

requirements development and

management activities have experience.

Strength

Activity performed 1 The project team performs its activities in

accordance with its documented

requirements development and

management plans.

There is no requirements development

and management plan, and so the

activities are not performed in accordance

with it.

Weakness

Activity performed 2 The project team develops, baselines,

and maintains software-related

contractual requirements and places them

under change control early in the project,

but not later than the release of the

solicitation package.

The project team developed and

baselined software-related requirements

before the solicitation package was

released. The project team provided no

evidence of change control.

Observation

Activity performed 3 The project team appraises system

requirements change requests for their

impact on the software being acquired.

The project team does not assess system

requirements change requests for their

impact on the software being acquired.

Weakness

Activity performed 4 The project team appraises changes to

the software-related contractual

requirements for their impact on

performance, architecture, supportability,

system resource utilization, and contract

schedule and cost.

The project team does not assess the

impact of software-related contractual

requirements changes on performance,

architecture, supportability, and resource

utilization. Cost and schedule impacts are

assessed.

Weakness

Activity performed 5 Bidirectional traceability between the

contractual requirements and the

contractor team’s software products and

services is maintained throughout the

effort.

The project team maintains traceability

between the contract deliverables and the

contractor’s task/subtask specification

numbering. Traceability to HUD’s original

requirements, however, is not maintained.

Observation

Activity performed 6 The end user and other affected groups

are involved in the development of all

software-related contractual requirements

and any subsequent change activity.

End users are involved in identifying

software-related requirements for

development but are not involved in

subsequent change activities.

Observation
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 19 GAO-01-962 HUD Information Systems

Public and Indian Housing Information Center system

Common feature Key practice Finding Rating

Measurement and analysis 1 Measurements are made and used to

determine the status of the requirements

development and management activities

and resultant products.

No measurements are made of HUD’s

requirements development and

management activities and resultant

products.

Weakness

Verifying implementation 1 Requirements development and

management activities are reviewed by

the acquisition organization on a periodic

basis.

Project status, including requirements

development and management activities,

is reviewed quarterly by acquisition

management.

Strength

Verifying implementation 2 Requirements development and

management activities are reviewed by

the project manager on both a periodic

and an event-driven basis.

The project manager is not briefed on

project activities and the project

manager’s reviews are not documented.

Weakness
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 20 GAO-01-962 HUD Information Systems

Table 4: Requirements Development and Management Findings for the Real Estate Management System

Real Estate Management System

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for establishing and

managing the software-related

contractual requirements.

HUD has no written policy for establishing

and managing software-related contractual

requirements.

Weakness

Commitment to perform 2 Responsibility has been designated for

requirements development and

management activities.

Responsibility for requirements

development and management activities is

designated in the project plan.

Strength

Ability to perform 1 A group is responsible for performing

requirements development and

management activities.

End users and project staff are responsible

for performing requirements development

and management activities.

Strength

Ability to perform 2 Adequate resources are provided for

requirements development and

management activities.

There is no mechanism to identify resource

needs and ensure that they are provided to

the project. The project manager said that

the team does not have adequate

resources for performing these activities.

Weakness

Ability to perform 3 Individuals performing requirements

development and management

activities have experience or receive

training.

Project team members performing

requirements development and

management activities have experience.

Strength

Activity performed 1 The project team performs its activities

in accordance with its documented

requirements development and

management plans.

There is no requirements development and

management plan, and so the activities are

not performed in accordance with it. The

team does follow a process for

requirements gathering.

Observation

Activity performed 2 The project team develops, baselines,

and maintains software-related

contractual requirements and places

them under change control early in the

project, but not later than the release

of the solicitation package.

The project team did not baseline

requirements before solicitation. Task

orders do not show traceability to softwarerelated requirements.

Weakness

Activity performed 3 The project team appraises system

requirements change requests for their

impact on the software being acquired.

The project team does not appraise system

requirements change requests for their

impact on the software being acquired.

Weakness

Activity performed 4 The project team appraises changes

to the software-related contractual

requirements for their impact on

performance, architecture, supportability, system resource utilization, and

contract schedule and cost.

The project team does not assess the

impact of software-related contractual

requirements changes on performance,

architecture, supportability, and resource

utilization. Cost and schedule are

assessed.

Weakness

Activity performed 5 Bidirectional traceability between the

contractual requirements and the

contractor team’s software products

and services is maintained throughout

the effort.

The project team does not maintain

bidirectional traceability between the

contractual requirements and contractor

work products.

Weakness
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 21 GAO-01-962 HUD Information Systems

Real Estate Management System

Common feature Key practice Finding Rating

Activity performed 6 The end user and other affected

groups are involved in the

development of all software-related

contractual requirements and any

subsequent change activity.

End users are involved in the development

of and changes to software-related

contractual requirements.

Strength

Measurement and analysis 1 Measurements are made and used to

determine the status of the

requirements development and

management activities and resultant

products.

No measurements are made of HUD’s

requirements development and

management activities and resultant

products.

Weakness

Verifying implementation 1 Requirements development and

management activities are reviewed

by the acquisition organization on a

periodic basis.

Requirements development and

management activities are not routinely

reviewed by the acquisition organization.

However, the project manager is a senior

official of the acquisition organization.

Observation

Verifying implementation 2 Requirements development and

management activities are reviewed

by the project manager on both a

periodic and an event-driven basis.

The project manager told us that he

participates in biweekly status meetings

that include a review of requirements

development and management issues. The

status meetings are documented.

Strength
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 22 GAO-01-962 HUD Information Systems

Table 5: Requirements Development and Management Findings for the Resident Assessment Subsystem

Resident Assessment Subsystem

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for establishing and

managing the software-related

contractual requirements.

HUD has no written policy for establishing and

managing software-related contractual

requirements. The project team used HUD’s

software development methodology as its

standard for establishing and managing

software-related contractual requirements.

Observation

Commitment to perform 2 Responsibility has been designated

for requirements development and

management activities.

Responsibility for requirements development

and management is designated in the project

plan.

Strength

Ability to perform 1 A group is responsible for performing

requirements development and

management activities.

The project team is responsible for performing

requirements development and management

activities.

Strength

Ability to perform 2 Adequate resources are provided for

requirements development and

management activities.

There is no mechanism to identify resource

needs and ensure that they are provided to the

project. However, the project manager stated

that they do have adequate resources for

performing requirements development and

management activities.

Strength

Ability to perform 3 Individuals performing requirements

development and management

activities have experience or receive

training.

Project team members performing

requirements development and management

activities have experience and have received

training.

Strength

Activity performed 1 The project team performs its

activities in accordance with its

documented requirements

development and management

plans.

The project team performs its activities in

accordance with its requirements development

and management plan.

Strength

Activity performed 2 The project team develops,

baselines, and maintains softwarerelated contractual requirements and

places them under change control

early in the project, but not later than

the release of the solicitation

package.

The project team developed and baselined

contract requirements in its functional

requirements document and requirements

traceability matrix, and placed the

requirements under change control.

Strength

Activity performed 3 The project team appraises system

requirements change requests for

their impact on the software being

acquired.

The project team assesses system

requirements change requests for their impact

on the software being acquired.

Strength

Activity performed 4 The project team appraises changes

to the software-related contractual

requirements for their impact on

performance, architecture, supportability, system resource utilization,

and contract schedule and cost.

The project team appraises changes to the

software-related contractual requirements for

impact on performance, architecture, supportability, system resource utilization, and

contract schedule and cost.

Strength
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 23 GAO-01-962 HUD Information Systems

Resident Assessment Subsystem

Common feature Key practice Finding Rating

Activity performed 5 Bidirectional traceability between the

contractual requirements and the

contractor team’s software products

and services is maintained

throughout the effort.

The project team maintains bidirectional

traceability between contractual requirements

and contractor work products.

Strength

Activity performed 6 The end user and other affected

groups are involved in the

development of all software-related

contractual requirements and any

subsequent change activity.

End users are involved in the development of

contractual requirements, generation of system

change requests, and testing.

Strength

Measurement and analysis 1 Measurements are made and used to

determine the status of the

requirements development and

management activities and resultant

products.

Masurements are only made of HUD’s

requirements development and management

products.

Observation

Verifying implementation 1 Requirements development and

management activities are reviewed

by the acquisition organization on a

periodic basis.

Requirements development and management

activities are not routinely reviewed by the

acquisition organization. However, cost and

schedule performance is reviewed periodically

by the acquisition organization.

Weakness

Verifying implementation 2 Requirements development and

management activities are reviewed

by the project manager on both a

periodic and an event-driven basis.

The project manager is not briefed on

requirements development and management

activities, and the project manager’s reviews

are not documented.

Weakness
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 24 GAO-01-962 HUD Information Systems

Table 6: Requirements Development and Management Findings for HUD’s Central Accounting and Program System

HUD’s Central Accounting and Program System

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for establishing and

managing the software-related

contractual requirements.

HUD has no written policy for establishing

and managing the software-related

contractual requirements. The project team

used HUD’s software development

methodology as its standard for establishing

and managing software-related contractual

requirements.

Observation

Commitment to perform 2 Responsibility has been designated for

requirements development and

management activities.

The project team has been designated

responsible for requirements development

and management, with no specific

designation of what group or person within

the team.

Observation

Ability to perform 1 A group is responsible for performing

requirements development and

management activities.

The project manager told us that the project

team is responsible for requirements

development and management activities.

Observation

Ability to perform 2 Adequate resources are provided for

requirements development and

management activities.

There is no mechanism to identify resource

needs and ensure that they are provided to

the project. The project team indicated that

they do not have adequate resources for

performing requirements development and

management activities.

Weakness

Ability to perform 3 Individuals performing requirements

development and management

activities have experience or receive

training.

Project team members performing

requirements development and

management activities have experience and

have received training.

Strength

Activity performed 1 The project team performs its activities

in accordance with its documented

requirements development and

management plans.

There is no requirements development and

management plan, and so the activities are

not performed in accordance with it.

Weakness

Activity performed 2 The project team develops, baselines,

and maintains software-related

contractual requirements and places

them under change control early in the

project, but not later than the release of

the solicitation package.

The project team developed and baselined

software-related contractual requirements

before the contractor began work, and has

maintained the requirements under

configuration board control.

Strength

Activity performed 3 The project team appraises system

requirements change requests for their

impact on the software being acquired.

The project team assesses all system

requirements change requests for their

impact on the software being acquired.

Strength

Activity performed 4 The project team appraises changes to

the software-related contractual

requirements for their impact on

performance, architecture, supportability, system resource utilization, and

contract schedule and cost.

The project team does not assess the

impact of software-related contractual

requirements changes on performance,

architecture, supportability, and system

resource utilization. Cost is appraised.

Weakness
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 25 GAO-01-962 HUD Information Systems

HUD’s Central Accounting and Program System

Common feature Key practice Finding Rating

Activity performed 5 Bidirectional traceability between the

contractual requirements and the

contractor team’s software products

and services is maintained throughout

the effort.

The project team maintains bidirectional

traceability between the contractual

requirements and the contractor team’s

software products and services.

Strength

Activity performed 6 The end user and other affected groups

are involved in the development of all

software-related contractual

requirements and any subsequent

change activity.

End users and an independent verification

and validation team were involved in testing

of changes. No evidence was provided to

document user activity in the development

of requirements.

Observation

Measurement and analysis 1 Measurements are made and used to

determine the status of the

requirements development and

management activities and resultant

products.

No measurements are made of HUD’s

requirements development and

management activities or resultant products.

Weakness

Verifying implementation 1 Requirements development and

management activities are reviewed by

the acquisition organization on a

periodic basis.

Acquisition management is regularly

updated on project status, including

requirements development and

maintenance activities.

Strength

Verifying implementation 2 Requirements development and

management activities are reviewed by

the project manager on both a periodic

and an event-driven basis.

Project manager has regular, documented

meetings to review requirements

development and management activities.

Strength
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 26 GAO-01-962 HUD Information Systems

Table 7: Requirements Development and Management Findings for the Empowerment Information System

Empowerment Information System

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for establishing and

managing the software-related

contractual requirements.

HUD has no written policy for

establishing and managing softwarerelated contractual requirements.

Weakness

Commitment to perform 2 Responsibility has been designated for

requirements development and

management activities.

Responsibility for requirements

development and management has not

been designated.

Weakness

Ability to perform 1 A group is responsible for performing

requirements development and

management activities.

The project team is responsible for

performing these activities.

Strength

Ability to perform 2 Adequate resources are provided for

requirements development and

management activities.

There is no mechanism to identify

resource needs and ensure that they

are provided to the project. The project

team stated that they do not have

adequate resources for performing

requirements development and

management activities.

Weakness

Ability to perform 3 Individuals performing requirements

development and management activities

have experience or receive training.

Project team members have training in

IT project management, including

requirements management.

Strength

Activity performed 1 The project team performs its activities

in accordance with its documented

requirements development and

management plans.

There is no requirements development

and management plan, and so the

activities are not performed in

accordance with it.

Weakness

Activity performed 2 The project team develops, baselines,

and maintains software-related

contractual requirements and places

them under change control early in the

project, but not later than the release of

the solicitation package.

The project team developed high-level

requirements. Detailed requirements

were developed by the contractor. No

evidence was provided regarding

change control.

Observation

Activity performed 3 The project team appraises system

requirements change requests for their

impact on the software being acquired.

The project team does not appraise

system requirements changes for their

impact on the software being acquired.

Weakness

Activity performed 4 The project team appraises changes to

the software-related contractual

requirements for their impact on

performance, architecture, supportability,

system resource utilization, and contract

schedule and cost.

The project team does not assess the

impact of software-related contractual

requirements changes on performance,

architecture, supportability, system

resource utilization, and contract

schedule and cost.

Weakness

Activity performed 5 Bidirectional traceability between the

contractual requirements and the

contractor team’s software products and

services is maintained throughout the

effort.

The project team does not maintain

bidirectional traceability between the

contractual requirements and contractor

work products.

Weakness
Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable

Page 27 GAO-01-962 HUD Information Systems

Empowerment Information System

Common feature Key practice Finding Rating

Activity performed 6 The end user and other affected groups

are involved in the development of all

software-related contractual

requirements and any subsequent

change activity.

No end users were involved in

requirements development or

subsequent change activities.

Weakness

Measurement and analysis 1 Measurements are made and used to

determine the status of the requirements

development and management activities

and resultant products.

Measurements are made that identify

the status of requirements development

and management activities.

Strength

Verifying implementation 1 Requirements development and

management activities are reviewed by

the acquisition organization on a periodic

basis.

Requirements development and

management activities are reviewed

periodically by the acquisition

organization.

Strength

Verifying implementation 2 Requirements development and

management activities are reviewed by

the project manager on both a periodic

and an event-driven basis.

The project manager told us that he is

not briefed on activities, and the project

manager’s reviews are not documented.

Weakness
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 28 GAO-01-962 HUD Information Systems

HUD did not meet the criteria for the repeatable level in the project

management key process area because of the number and severity of key

practice weaknesses found. These weaknesses increase the risk that the

software acquisition project, the project team, and supporting contractors

will not be adequately managed, resulting in software that does not meet

mission needs or acquisitions that do not meet cost, schedule, and

performance goals.

The purpose of project management is to direct and oversee the activities

of the project team and supporting contractors to ensure a timely,

efficient, and effective software acquisition. Within this key process area,

the SA-CMM specifies key practices for each of the five common features

that an organization must implement effectively to achieve the repeatable

level of maturity. Generally, these practices include having a project team

organized to accomplish the project’s objective; having a written policy for

the management of the project; documenting plans for the activities of the

project team; measuring and controlling the cost, schedule, and

performance objectives throughout the software acquisition; and reporting

to management periodically on the status of project management

activities.

Of the 80 key practices for this key process area, HUD had 45 strengths, 25

weaknesses, and 10 observations. For the

• Commitment to perform feature, there were four strengths and six

weaknesses.

• Ability to perform feature, there were 14 strengths, four weaknesses, and

two observations.

• Activities performed feature, there were 22 strengths, seven weaknesses,

and six observations.

• Measurement and analysis feature, there were no strengths, four

weaknesses, and one observation.

• Verifying implementation feature, there were five strengths, four

weaknesses, and one observation.

In addition, none of the projects had strengths in all the key practices.

As a result of these weaknesses, HUD is exposed to increased risks that

software acquisition projects will exceed cost, schedule, and performance

goals, and that acquired software will not meet mission needs. HUD

acknowledges the need to improve its project management processes and

is committed to improving its software and system acquisition capability.

The department has prepared a statement of work to obtain contract

Chapter 3: HUD’s Project Management

Processes Are Not Repeatable
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 29 GAO-01-962 HUD Information Systems

support to begin a software process improvement effort. This document

specifies that the contractor would review HUD software acquisition

processes, prioritize the key processes and practices to address, and

develop a plan to improve HUD’s acquisition processes.

Details of our evaluation are provided in the following figure and tables.

Figure 3 provides a comprehensive listing of the strengths, weaknesses,

and observations for the project management key process area. The

specific findings supporting the practice ratings cited in figure 3 are in

tables 8 through 12.
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 30 GAO-01-962 HUD Information Systems

Figure 3: Project Management Summary

Common Projects

features Key practices PIC REMS RASS HUDCAPS EIS

Commitment to

perform 1

The acquisition organization has a written policy for

management of the software acquisition project. Weakness Weakness Weakness Weakness Weakness

Commitment to

perform 2

Responsibility has been designated for project

management activities. Weakness Strength Strength Strength Strength

Ability to

perform 1

A group is responsible for managing software

acquisition activities. Observation Strength Strength Strength Strength

Ability to

perform 2

Adequate resources for the project team are provided

for the duration of the project. Weakness Weakness Strength Weakness Weakness

Ability to

perform 3

When project trade-offs are necessary, the project

manager is permitted to alter the software acquisition

cost, schedule, or performance.

Strength Strength Strength Strength Strength

Ability to

perform 4

Individuals performing software acquisition

management activities have experience or receive

training.

Strength Observation Strength Strength Strength

Activity

performed 1

The project team performs its activities in accordance

with its documented software acquisition plans. Observation Strength Strength Strength Strength

Activity

performed 2

The roles, responsibilities, and authority of the project

function are documented, maintained, and

communicated to affected groups.

Weakness Observation Strength Strength Strength

Activity

performed 3

The project team’s commitments, and changes to

commitments, are communicated to affected groups. Observation Strength Strength Strength Strength

Activity

performed 4

The project team tracks the cost, schedule, resource,

and technical risks of the project. Observation Weakness Strength Weakness Strength

Activity

performed 5

The project team tracks project issues, status,

execution, funding, and expenditures against project

plans and takes action.

Observation Strength Strength Strength Strength

Activity

performed 6

The project team implements a corrective action system

for the identification, recording, tracking, and correction

of problems discovered during the software acquisition.

Weakness Observation Strength Strength Strength

Activity

performed 7

The project team keeps its plans current during the life

of the project as replanning occurs, issues are resolved,

and new risks are discovered.

Weakness Weakness Strength Weakness Strength

Measurement

and analysis 1

Measurements are made and used to determine the

status of the project management activities and

resultant products.

Weakness Weakness Observation Weakness Weakness

Verifying

implementation 1

Project management activities are reviewed by the

acquisition organization on a periodic basis. Strength Weakness Strength Strength Strength

Verifying

implementation 2

Project management activities are reviewed by the

project manager on both a periodic and an event-driven

basis.

Weakness Observation Weakness Strength Weakness

Key Strength Strength: Key practice effectively implemented.

Weakness Weakness: Key practice not effectively implemented.

Observation Observation: Key practice evaluated; evidence not sufficient to rate as a strength, or practice only partially performed.
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 31 GAO-01-962 HUD Information Systems

Table 8: Project Management Findings for the Public and Indian Housing Information Center System

Public and Indian Housing Information Center system

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a written

policy for management of the software

acquisition project.

HUD has no written policy for managing

the acquisition of software products and

services.

Weakness

Commitment to perform 2 Responsibility has been designated for

project management activities.

The project team told us that project

management responsibilities had been

assigned, but no documentation was

provided to support this statement.

Weakness

Ability to perform 1 A group is responsible for managing

software acquisition activities.

The project team told us that IT and

business project leads and support staff

are responsible for managing software

acquisition activities, but no documentation

was provided to support this statement.

Observation

Ability to perform 2 Adequate resources for the project team

are provided for the duration of the

project.

There is no mechanism to identify resource

needs and ensure that they are provided to

the project. Team members indicated that

the team does not have adequate

resources for conducting project

management activities.

Weakness

Ability to perform 3 When project trade-offs are necessary,

the project manager is permitted to alter

the software acquisition cost, schedule, or

performance.

The project manager is permitted to alter

schedule or performance to ensure that

annual project budgets are not exceeded.

Strength

Ability to perform 4 Individuals performing software

acquisition management activities have

experience or receive training.

Project team members performing project

management activities have experience

and project management training.

Strength

Activity performed 1 The project team performs its activities in

accordance with its documented software

acquisition plans.

The project team told us they did develop

project management plans, but no

documentation was provided to support

this statement.

Observation

Activity performed 2 The roles, responsibilities, and authority

of the project function are documented,

maintained, and communicated to

affected groups.

The project team has not documented the

roles, responsibilities, and authority of its

members.

Weakness

Activity performed 3 The project team’s commitments, and

changes to commitments, are

communicated to affected groups.

The commitments and changes to

commitments are communicated during

small informal group meetings, but they are

not documented and no records are kept.

Observation

Activity performed 4 The project team tracks the cost,

schedule, resource, and technical risks of

the project.

Cost and schedule are tracked, but

resource and technical risks of the project

are not tracked.

Observation

Activity performed 5 The project team tracks project issues,

status, execution, funding, and

expenditures against project plans and

takes action.

The project team only tracks and takes

action on changes to the project schedule.

Observation
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 32 GAO-01-962 HUD Information Systems

Public and Indian Housing Information Center system

Common feature Key practice Finding Rating

Activity performed 6 The project team implements a corrective

action system for the identification,

recording, tracking, and correction of

problems discovered during the software

acquisition.

The project team does not maintain a

corrective action system for the

identification, recording, tracking, and

correction of problems discovered during

the software acquisition.

Weakness

Activity performed 7 The project team keeps its plans current

during the life of the project as replanning

occurs, issues are resolved, and new

risks are discovered.

The project team does not keep its plans

up to date over the life of the project.

Weakness

Measurement and analysis 1 Measurements are made and used to

determine the status of the project

management activities and resultant

products.

No measurements are made of HUD’s

project management activities or resultant

products.

Weakness

Verifying implementation 1 Project management activities are

reviewed by the acquisition organization

on a periodic basis.

Acquisition management reviews project

activities quarterly.

Strength

Verifying implementation 2 Project management activities are

reviewed by the project manager on both

a periodic and an event-driven basis.

The project manager stated that while he

meets with members of his staff daily to

discuss the status of the project, meeting

results and action items are not

documented.

Weakness
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 33 GAO-01-962 HUD Information Systems

Table 9: Project Management Findings for the Real Estate Management System

Real Estate Management System

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for management of the

software acquisition project.

HUD has no written policy for managing the

acquisition of software products and

services.

Weakness

Commitment to perform 2 Responsibility has been designated for

project management activities.

Responsibility for project management

activities has been assigned to the project

manager in the project management plan.

Strength

Ability to perform 1 A group is responsible for managing

software acquisition activities.

The project manager, along with business

and IT support staff, is responsible for

project management activities.

Strength

Ability to perform 2 Adequate resources for the project

team are provided for the duration of

the project.

There is no mechanism to identify resource

needs and ensure that they are provided to

the project. The project manager stated that

the team does not have adequate resources

for conducting project management

activities.

Weakness

Ability to perform 3 When project trade-offs are

necessary, the project manager is

permitted to alter the software

acquisition cost, schedule, or

performance.

The project manager is permitted to alter

schedule or performance to ensure that

annual project budgets are not exceeded.

Strength

Ability to perform 4 Individuals performing software

acquisition management activities

have experience or receive training.

The project team told us that individuals

performing project management activities

have experience and have had training.

However, the team did not provide

documentation to support this statement.

Observation

Activity performed 1 The project team performs its activities

in accordance with its documented

software acquisition plans.

Software acquisition project management

plans are used to guide the activities of the

project team.

Strength

Activity performed 2 The roles, responsibilities, and

authority of the project function are

documented, maintained, and

communicated to affected groups.

Roles and responsibilities are defined in the

project plan; however, responsible team

members are not named, which hinders

effective communication of this information.

Observation

Activity performed 3 The project team’s commitments, and

changes to commitments, are

communicated to affected groups.

The commitments, and changes to

commitments, are communicated to affected

groups through weekly status meetings,

emails, and telephone conversations.

Strength

Activity performed 4 The project team tracks the cost,

schedule, resource, and technical

risks of the project.

The project team does not track risks

related to cost, schedule, resource, and

technical aspects of the project.

Weakness

Activity performed 5 The project team tracks project issues,

status, execution, funding, and

expenditures against project plans and

takes action.

The project manager tracks project status,

execution, funding, and expenditures, and

changes the scope or schedule if

necessary.

Strength
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 34 GAO-01-962 HUD Information Systems

Real Estate Management System

Common feature Key practice Finding Rating

Activity performed 6 The project team implements a

corrective action system for the

identification, recording, tracking, and

correction of problems discovered

during the software acquisition.

The project team does not maintain a

corrective action system for the

identification, recording, tracking, and

correction of problems discovered during

the software acquisition. Minutes of

biweekly status meetings identify action

items and who is responsible for each item.

Observation

Activity performed 7 The project team keeps its plans

current during the life of the project as

replanning occurs, issues are

resolved, and new risks are

discovered.

The project team does not keep its plans up

to date over the life of the project.

Weakness

Measurement and analysis 1 Measurements are made and used to

determine the status of the project

management activities and resultant

products.

No measurements are made of HUD’s

project management activities or resultant

products.

Weakness

Verifying implementation 1 Project management activities are

reviewed by the acquisition

organization on a periodic basis.

Acquisition management does not regularly

review project management activities.

Weakness

Verifying implementation 2 Project management activities are

reviewed by the project manager on

both a periodic and an event-driven

basis.

The project manager is involved with the

project, and status meetings are held

regularly; however, the project manager’s

reviews are not documented.

Observation
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 35 GAO-01-962 HUD Information Systems

Table 10: Project Management Findings for the Resident Assessment Subsystem

Resident Assessment Subsystem

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for management of the

software acquisition project.

HUD has no written policy for managing the

acquisition of software products and

services. The project team uses HUD’s

software development methodology where

applicable.

Weakness

Commitment to perform 2 Responsibility has been designated for

project management activities.

Responsibility for project management

activities has been assigned to the IT and

business project leads.

Strength

Ability to perform 1 A group is responsible for managing

software acquisition activities.

The IT and business project leads and

support staff are responsible for project

management activities.

Strength

Ability to perform 2 Adequate resources for the project

team are provided for the duration of

the project.

There is no mechanism to identify resource

needs and ensure that they are provided to

the project. The project manager stated that

the team has adequate resources for

conducting project management activities.

Strength

Ability to perform 3 When project trade-offs are necessary,

the project manager is permitted to alter

the software acquisition cost, schedule,

or performance.

The project manager is permitted to alter

schedule or performance to ensure that

annual project budgets are not exceeded.

Strength

Ability to perform 4 Individuals performing software

acquisition management activities have

experience or receive training.

Project team members performing project

management activities have experience and

have had training.

Strength

Activity performed 1 The project team performs its activities

in accordance with its documented

software acquisition plans.

Software acquisition project management

plans are used to guide the activities of the

project team.

Strength

Activity performed 2 The roles, responsibilities, and authority

of the project function are documented,

maintained, and communicated to

affected groups.

Roles, responsibilities, and authority are

defined in the project plan and in the

Configuration Control Board’s (CCB) role

matrix.

Strength

Activity performed 3 The project team’s commitments, and

changes to commitments, are

communicated to affected groups.

The project team’s commitments and

changes to commitments regarding this

project are communicated to affected

groups.

Strength

Activity performed 4 The project team tracks the cost,

schedule, resource, and technical risks

of the project.

A risk management plan is maintained that

tracks risks associated with costs, schedule,

resource, and technical risks.

Strength

Activity performed 5 The project team tracks project issues,

status, execution, funding, and

expenditures against project plans and

takes action.

The project team uses contractor weekly

and monthly status reports to track

contractor costs and schedule issues. Also,

a quarterly project control review is done to

review cost and schedule status.

Strength

Activity performed 6 The project team implements a

corrective action system for the

identification, recording, tracking, and

correction of problems discovered

during the software acquisition.

A corrective action system has been

implemented for the identification, recording,

tracking, and correction of problems

discovered during the software acquisition.

Strength
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 36 GAO-01-962 HUD Information Systems

Resident Assessment Subsystem

Common feature Key practice Finding Rating

Activity performed 7 The project team keeps its plans

current during the life of the project as

replanning occurs, issues are resolved,

and new risks are discovered.

The project plans are kept up to date as

replanning occurs, issues are resolved, and

new risks are discovered. Monthly reports

are provided that show changes to

scheduled software releases resulting from

replanning.

Strength

Measurement and analysis 1 Measurements are made and used to

determine the status of the project

management activities and resultant

products.

Measurements are only made of HUD’s

project management products.

Observation

Verifying implementation 1 Project management activities are

reviewed by the acquisition

organization on a periodic basis.

The acquisition organization reviews project

management activities on a quarterly basis.

Strength

Verifying implementation 2 Project management activities are

reviewed by the project manager on

both a periodic and an event-driven

basis.

The project manager stated that while he

meets with members of his staff daily to

discuss the status of the project, meeting

results and action items are not

documented.

Weakness
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 37 GAO-01-962 HUD Information Systems

Table 11: Project Management Findings for HUD’s Central Accounting and Program System

HUD’s Central Accounting and Program System

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for management of the

software acquisition project.

HUD has no written policy for managing the

acquisition of software products and services.

Weakness

Commitment to perform 2 Responsibility has been designated

for project management activities.

Responsibility for project management activities

has been assigned to the project lead.

Strength

Ability to perform 1 A group is responsible for managing

software acquisition activities.

The IT and business project leads are

responsible for managing software acquisition

activities.

Strength

Ability to perform 2 Adequate resources for the project

team are provided for the duration of

the project.

There is no mechanism to identify resource

needs and ensure they were made available to

the project. The project manager stated that the

team does not have adequate resources for

performing project management activities.

Weakness

Ability to perform 3 When project trade-offs are

necessary, the project manager is

permitted to alter the software

acquisition cost, schedule, or

performance.

The project manager is permitted to alter the

schedule or performance to ensure that annual

project budgets are not exceeded.

Strength

Ability to perform 4 Individuals performing software

acquisition management activities

have experience or receive training.

Project team members performing project

management activities have experience and

have had training.

Strength

Activity performed 1 The project team performs its

activities in accordance with its

documented software acquisition

plans.

Software acquisition project management plans

are used to guide the activities of the project

team.

Strength

Activity performed 2 The roles, responsibilities, and

authority of the project function are

documented, maintained, and

communicated to affected groups.

The roles, responsibilities, and authority of the

project team are defined in the software

acquisition project plan.

Strength

Activity performed 3 The project team’s commitments, and

changes to commitments, are

communicated to affected groups.

The commitments and changes to

commitments are communicated to affected

groups through weekly status meetings, emails,

and telephone conversations.

Strength

Activity performed 4 The project team tracks the cost,

schedule, resource, and technical

risks of the project.

The project plans and exhibits identify risks

associated with the project and define

mitigation plans for these risks. However, the

project team does not track the risks.

Weakness

Activity performed 5 The project team tracks project

issues, status, execution, funding,

and expenditures against project

plans and takes action.

The project team tracks project issues, status,

execution, and funding through weekly

functional and technical status meetings.

Funding is tracked and actual costs compared

to projected costs.

Strength

Activity performed 6 The project team implements a

corrective action system for the

identification, recording, tracking, and

correction of problems discovered

during the software acquisition.

A corrective action system has been

implemented for the identification, recording,

tracking, and correction of problems discovered

during the software acquisition.

Strength
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 38 GAO-01-962 HUD Information Systems

HUD’s Central Accounting and Program System

Common feature Key practice Finding Rating

Activity performed 7 The project team keeps its plans

current during the life of the project

as replanning occurs, issues are

resolved, and new risks are

discovered.

The project team does have a risk assessment

and mitigation strategy. However, the risks are

not tracked, and plans are not kept current.

Weakness

Measurement and analysis 1 Measurements are made and used to

determine the status of the project

management activities and resultant

products.

No measurements are made of HUD’s project

management activities or resultant products.

Weakness

Verifying implementation 1 Project management activities are

reviewed by the acquisition

organization on a periodic basis.

The acquisition organization reviews project

management activities on a regular basis

through meetings with the project manager.

Strength

Verifying implementation 2 Project management activities are

reviewed by the project manager on

both a periodic and an event-driven

basis.

The project manager reviews project

management activities regularly through

briefings and document review.

Strength
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 39 GAO-01-962 HUD Information Systems

Table 12: Project Management Findings for the Empowerment Information System

Empowerment Information System

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for management of the

software acquisition project.

HUD has no written policy for managing the

acquisition of software products and services.

Weakness

Commitment to perform 2 Responsibility has been designated

for project management activities.

Responsibility for project management activities

has been assigned to the project technical lead.

Strength

Ability to perform 1 A group is responsible for managing

software acquisition activities.

The project lead and assigned staff are

responsible for performing project management

activities.

Strength

Ability to perform 2 Adequate resources for the project

team are provided for the duration of

the project.

There is no mechanism to identify resource

needs and ensure that they are made available

to the project. The project manager stated that

the team does not have adequate resources for

performing project management activities.

Weakness

Ability to perform 3 When project trade-offs are

necessary, the project manager is

permitted to alter the software

acquisition cost, schedule, or

performance.

The project manager is allowed to alter

schedule or performance to ensure that annual

project budgets are not exceeded.

Strength

Ability to perform 4 Individuals performing software

acquisition management activities

have experience or receive training.

Project team members performing project

management activities have experience and

project management training.

Strength

Activity performed 1 The project team performs its

activities in accordance with its

documented software acquisition

plans.

Software acquisition project management plans

are used to guide the activities of the project

team.

Strength

Activity performed 2 The roles, responsibilities, and

authority of the project function are

documented, maintained, and

communicated to affected groups.

Roles, responsibilities, and authority of the

project team are defined in the project

organization chart.

Strength

Activity performed 3 The project team’s commitments,

and changes to commitments, are

communicated to affected groups.

The project manager uses contractor status

reports to communicate commitments and

changes to commitments to the project team

and other affected groups in meetings.

Strength

Activity performed 4 The project team tracks the cost,

schedule, resource, and technical

risks of the project.

The project team does assess and track risks

for several areas, including contract, funding,

schedule, management, and technical areas.

Strength

Activity performed 5 The project team tracks project

issues, status, execution, funding,

and expenditures against project

plans and takes action.

The project team did conduct a review of the

software developed by the contractor. As a

result of issues found during that review,

modifications were made to the contract.

Strength

Activity performed 6 The project team implements a

corrective action system for the

identification, recording, tracking,

and correction of problems

discovered during the software

acquisition.

A remedial action plan was developed and a

corrective action system implemented to

record, track, and correct problems found

during the software acquisition.

Strength
Chapter 3: HUD’s Project Management

Processes Are Not Repeatable

Page 40 GAO-01-962 HUD Information Systems

Empowerment Information System

Common feature Key practice Finding Rating

Activity performed 7 The project team keeps its plans

current during the life of the project

as replanning occurs, issues are

resolved, and new risks are

discovered.

The replanning is documented in the remedial

action plan.

Strength

Measurement and analysis 1 Measurements are made and used

to determine the status of the

project management activities and

resultant products.

No measurements are made of HUD’s project

management activities and resultant products.

Weakness

Verifying implementation 1 Project management activities are

reviewed by the acquisition

organization on a periodic basis.

The acquisition organization reviews project

management activities quarterly.

Strength

Verifying implementation 2 Project management activities are

reviewed by the project manager on

both a periodic and an event-driven

basis.

The project manager is not briefed by other

members on either a regular or event-driven

basis, and project manager reviews are not

documented.

Weakness
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 41 GAO-01-962 HUD Information Systems

HUD’s contract tracking and oversight processes do not satisfy the criteria

for the repeatable level of maturity because of the number and severity of

key practice weaknesses found. These weaknesses increase the risk that

software activities will not be performed in accordance with contractual

requirements, resulting in software acquisitions being late, costing more

than intended, and not performing as intended.

The purpose of contract tracking and oversight is to ensure that the

software activities under contract are being performed in accordance with

contractual requirements. Within this key process area, the SA-CMM

specifies a number of key practices across the five common features that

an organization must implement effectively to achieve the repeatable level

of maturity. Generally, these practices include having a written

organizational policy for contract tracking and oversight, documenting

plans for contract tracking and oversight activities, reviewing contractor

software planning documents and using those documents to oversee the

contractor’s efforts, comparing actual cost and schedule to planned

budgets and schedules, and having a mechanism to ensure that contractordelivered work products meet contract requirements.

For the 85 key practices in this key process area, HUD had 45 strengths, 33

weaknesses, four observations, and three practices not rated. For the

• Commitment to perform feature, there were 12 strengths and three

weaknesses.

• Ability to perform feature, there were ten strengths and five weaknesses.

• Activities performed feature, there were 20 strengths, 15 weaknesses, two

observations, and three practices not rated.

• Measurement and analysis feature, there were no strengths and five

weaknesses.

• Verifying implementation feature, there were three strengths, five

weaknesses, and two observations.

In addition, none of the projects had strengths in all the key practices.

As a result of these weaknesses, HUD is exposed to increased risk that

software activities will not be performed in accordance with contractual

requirements, resulting in software acquisitions being late, costing more

than planned, and not performing as intended. HUD acknowledges the

need to improve its contract tracking and oversight processes and is

committed to improving its software and system acquisition capability.

The department has prepared a statement of work to obtain contractor

support to begin a software process improvement effort. This document

Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 42 GAO-01-962 HUD Information Systems

specifies that the contractor would review HUD software acquisition

processes, prioritize the key processes and practices to address, and

develop a plan to improve HUD’s acquisition processes.

Details of our evaluation are provided in the following figure and tables.

Figure 4 provides a comprehensive listing of the strengths, weaknesses,

and observations for the contract tracking and oversight key process area.

The specific findings supporting the practice ratings cited in figure 4 are in

tables 13 through 17.
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 43 GAO-01-962 HUD Information Systems

Figure 4: Contract Tracking and Oversight Summary

Common Projects

features Key practices PIC REMS RASS HUDCAPS EIS

Commitment to

perform 1

The acquisition organization has a written policy for

managing the contract tracking and oversight of the

software acquisition project.

Weakness Strength Strength Strength Strength

Commitment to

perform 2

Responsibility has been designated for contract tracking

and oversight activities. Weakness Strength Strength Strength Strength

Commitment to

perform 3

The project team includes contracting specialists in the

management of the contract. Weakness Strength Strength Strength Strength

Ability to

perform 1

A group is responsible for managing contract tracking

and oversight activities. Weakness Strength Strength Strength Strength

Ability to

perform 2

Adequate resources are provided for contract tracking

and oversight activities. Weakness Weakness Strength Weakness Weakness

Ability to

perform 3

Individuals performing contract tracking and oversight

activities have experience or receive training. Strength Strength Strength Strength Strength

Activity

performed 1

The project team performs its activities in accordance

with documented contract tracking and oversight plans. Weakness Weakness Strength Weakness Weakness

Activity

performed 2

The project team reviews required contractor software

planning documents which, when satisfactory, are used

to oversee the contractor team’s software effort.

Weakness Weakness Weakness Weakness Weakness

Activity

performed 3

The project team conducts periodic reviews and

interchanges with the contractor team. Strength Strength Strength Strength Strength

Activity

performed 4

The actual cost and schedule of the contractor’s

software effort are compared to planned budgets and

schedules, and issues are identified.

Strength Strength Strength Observation Strength

Activity

performed 5

The size, critical computer resources, and technical

activities associated with the contractor team’s work

products are tracked, and issues are identified.

Weakness Weakness Strength Strength Observation

Activity

performed 6

The project team reviews and tracks the development of

the software engineering environment required to

provide life-cycle support for the acquired software, and

issues are identified.

Strength Strength Strength Strength Strength

Activity

performed 7

Any issues found by the project team during contract

tracking and oversight activities are recorded in the

appropriate corrective action system, action is taken,

and issues are tracked to closure.

Weakness Weakness Strength Weakness Strength

Activity

performed 8

The project team ensures that changes to the softwarerelated contractual requirements are coordinated with

affected groups and individuals, such as the contracting

official, contractor, and end user.

Strength Not rated Not rated Not rated Weakness

Measurement

and analysis 1

Measurements are made and used to determine the

status of the contract tracking and oversight activities

and resultant products.

Weakness Weakness Weakness Weakness Weakness

Verifying

implementation 1

Contract tracking and oversight activities are reviewed

by acquisition management on a periodic basis. Weakness Weakness Weakness Strength Strength

Verifying

implementation 2

Contract tracking and oversight activities are reviewed

by the project manager on both a periodic and an eventdriven basis.

Weakness Observation Weakness Strength Observation

Key Strength Strength: Key practice effectively implemented.

Weakness Weakness: Key practice not effectively implemented.

Observation Observation: Key practice evaluated; evidence not sufficient to rate as a strength, or practice only partially performed.
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 44 GAO-01-962 HUD Information Systems

Table 13: Contract Tracking and Oversight Findings for the Public and Indian Housing Information Center System

Public and Indian Housing Information Center system

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for managing the contract

tracking and oversight of the software

acquisition project.

HUD has no written policy for contract

tracking and oversight of the acquisition of

software products and services, and the

project team did not establish one.

Weakness

Commitment to perform 2 Responsibility has been designated for

contract tracking and oversight

activities.

Responsibility for contract tracking and

oversight activities is not designated.

Weakness

Commitment to perform 3 The project team includes contracting

specialists in the management of the

contract.

The project team told us they did include

contracting specialists in the management

of the contract, but no documentation was

provided to support this statement.

Weakness

Ability to perform 1 A group is responsible for managing

contract tracking and oversight

activities.

The project team told us there was a group

assigned responsibility for contract tracking

and oversight activities, but no

documentation was provided to support

this statement.

Weakness

Ability to perform 2 Adequate resources are provided for

contract tracking and oversight

activities.

There is no mechanism to identify resource

needs and ensure that they are provided to

the project. Team members indicated that

the resources for performing contract

tracking and oversight activities are not

adequate.

Weakness

Ability to perform 3 Individuals performing contract tracking

and oversight activities have

experience or receive training.

Project team members performing contract

tracking and oversight activities have

experience.

Strength

Activity performed 1 The project team performs its activities

in accordance with its documented

contract tracking and oversight plans.

There is no contract tracking and oversight

plan; therefore, the activities are not

performed in accordance with it.

Weakness

Activity performed 2 The project team reviews required

contractor software planning

documents which, when satisfactory,

are used to oversee the contractor

team’s software effort.

The contractor did not prepare software

planning documents that the project team

could use to oversee the contractor’s

software development effort.

Weakness

Activity performed 3 The project team conducts periodic

reviews and interchanges with the

contractor team.

The project team does have periodic

reviews and discussions with the

contractor team that are documented.

Strength

Activity performed 4 The actual cost and schedule of the

contractor’s software effort are

compared to planned budgets and

schedules, and issues are identified.

The project team does compare the actual

cost and schedule of the contractor’s

software engineering effort to planned

schedules and budgets, identifies issues,

and documents results.

Strength

Activity performed 5 The size, critical computer resources,

and technical activities associated with

the contractor team’s work products are

tracked, and issues are identified.

The project team does not track the size,

critical computer resources, and technical

activities associated with the contractor

team’s work products or identify issues.

Weakness
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 45 GAO-01-962 HUD Information Systems

Public and Indian Housing Information Center system

Common feature Key practice Finding Rating

Activity performed 6 The project team reviews and tracks

the development of the software

engineering environment required to

provide life-cycle support for the

acquired software, and issues are

identified.

The project team does review and track

development of the software engineering

environment required to provide life-cycle

support for the acquired software, and

issues are identified and documented.

Strength

Activity performed 7 Any issues found by the project team

during contract tracking and oversight

activities are recorded in the

appropriate corrective action system,

action is taken, and issues are tracked

to closure.

The project team does not have a

corrective action system to record or track

action taken on issues found by the project

team during contract tracking and oversight

activities.

Weakness

Activity performed 8 The project team ensures that changes

to the software-related contractual

requirements are coordinated with

affected groups and individuals, such

as the contracting official, contractor,

and end user.

The project team does ensure that

changes to contractual requirements are

coordinated with all affected groups and

individuals.

Strength

Measurement and analysis 1 Measurements are made and used to

determine the status of the contract

tracking and oversight activities and

resultant products.

No measurements are made of HUD’s

contract tracking and oversight activities or

resultant products.

Weakness

Verifying implementation 1 Contract tracking and oversight

activities are reviewed by acquisition

management on a periodic basis.

The acquisition organization does not

review contract tracking and oversight

activities on a periodic basis.

Weakness

Verifying implementation 2 Contract tracking and oversight

activities are reviewed by the project

manager on both a periodic and an

event-driven basis.

The project manager stated that while he

discusses the status of the project with

team members, meeting results and action

items are not documented.

Weakness
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 46 GAO-01-962 HUD Information Systems

Table 14: Contract Tracking and Oversight Findings for the Real Estate Management System

Real Estate Management System

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for managing the

contract tracking and oversight of the

software acquisition project.

HUD has no written policy for contract

tracking and oversight of the acquisition of

software products and services. The

project team is following the policy in

HUD’s procurement manual for contract

tracking and oversight.

Strength

Commitment to perform 2 Responsibility has been designated for

contract tracking and oversight

activities.

Responsibility for contract tracking and

oversight activities has been designated to

contract specialists on the team.

Strength

Commitment to perform 3 The project team includes contracting

specialists in the management of the

contract.

The project team includes contract

specialists to help manage the contract.

Strength

Ability to perform 1 A group is responsible for managing

contract tracking and oversight

activities.

A group is assigned responsibility for

contract tracking and oversight activities.

Strength

Ability to perform 2 Adequate resources are provided for

contract tracking and oversight

activities.

There is no mechanism to identify resource

needs and ensure that they are provided to

the project. Team members indicated that

the resources for performing contract

tracking and oversight activities are not

adequate.

Weakness

Ability to perform 3 Individuals performing contract

tracking and oversight activities have

experience or receive training.

Project team members performing contract

tracking and oversight activities have

experience and have received training.

Strength

Activity performed 1 The project team performs its activities

in accordance with its documented

contract tracking and oversight plans.

There is no contract tracking and oversight

plan; therefore, the activities are not

performed in accordance with it.

Weakness

Activity performed 2 The project team reviews required

contractor software planning

documents which, when satisfactory,

are used to oversee the contractor

team’s software effort.

The contractor did prepare software

planning documents that could be used to

oversee the contractor team’s software

development effort, but no evidence was

provided that these were reviewed and/or

approved by the project team.

Weakness

Activity performed 3 The project team conducts periodic

reviews and interchanges with the

contractor team.

The project team does have periodic

reviews and discussions with the

contractor team that are documented.

Strength

Activity performed 4 The actual cost and schedule of the

contractor’s software effort are

compared to planned budgets and

schedules, and issues are identified.

The project team does compare the actual

cost and schedule of the contractor’s

software engineering effort to planned

schedules and budgets, identify issues,

and document results.

Strength

Activity performed 5 The size, critical computer resources,

and technical activities associated with

the contractor team’s work products

are tracked, and issues are identified.

The project team told us they track the

size, critical computer resources, and

technical activities associated with the

contractor team’s work products, but no

documentation was provided to support

this statement.

Weakness
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 47 GAO-01-962 HUD Information Systems

Real Estate Management System

Common feature Key practice Finding Rating

Activity performed 6 The project team reviews and tracks

the development of the software

engineering environment required to

provide life-cycle support for the

acquired software, and issues are

identified.

The project team does review and track the

development of the software engineering

environment required to provide life-cycle

support for the acquired software, and

issues are identified and documented.

Strength

Activity performed 7 Any issues found by the project team

during contract tracking and oversight

activities are recorded in the

appropriate corrective action system,

action is taken, and issues are tracked

to closure.

The project team does not have a

corrective action system to record or track

action taken on issues found by the project

team during contract tracking and oversight

activities.

Weakness

Activity performed 8 The project team ensures that

changes to the software-related

contractual requirements are

coordinated with affected groups and

individuals, such as the contracting

official, contractor, and end user.

The project team stated that no changes to

the software-related contractual

requirements have taken place because

their contracts are broad enough to cover

changes to the contractual requirements.

Not rated

Measurement and analysis 1 Measurements are made and used to

determine the status of the contract

tracking and oversight activities and

resultant products.

No measurements are made of HUD’s

contract tracking and oversight activities or

resultant products.

Weakness

Verifying implementation 1 Contract tracking and oversight

activities are reviewed by acquisition

management on a periodic basis.

The acquisition organization does not

review contract tracking and oversight

activities on a periodic basis.

Weakness

Verifying implementation 2 Contract tracking and oversight

activities are reviewed by the project

manager on both a periodic and an

event-driven basis.

The project manager informally reviews the

work of the project team; however, the

results of these reviews are not

documented. Minutes of biweekly status

meetings with contractors are maintained.

Observation
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 48 GAO-01-962 HUD Information Systems

Table 15: Contract Tracking and Oversight Findings for the Resident Assessment Subsystem

Resident Assessment Subsystem

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for managing the

contract tracking and oversight of the

software acquisition project.

HUD has no written policy for contract

tracking and oversight of the acquisition of

software products and services. The

project team is following the policy in

HUD’s Procurement Handbook for contract

tracking and oversight.

Strength

Commitment to perform 2 Responsibility has been designated for

contract tracking and oversight

activities.

Responsibility for contract tracking and

oversight activities is designated in the

project plan.

Strength

Commitment to perform 3 The project team includes contracting

specialists in the management of the

contract.

The project team includes contract

specialists to help manage the contract.

Strength

Ability to perform 1 A group is responsible for managing

contract tracking and oversight

activities.

A group is assigned responsibility for

contract tracking and oversight activities.

Strength

Ability to perform 2 Adequate resources are provided for

contract tracking and oversight

activities.

There is no mechanism to identify resource

needs and ensure that they are provided to

the project. The project manager stated

that the team does have adequate

resources for performing contract tracking

and oversight activities.

Strength

Ability to perform 3 Individuals performing contract

tracking and oversight activities have

experience or receive training.

Project team members performing contract

tracking and oversight activities have

experience and have received training.

Strength

Activity performed 1 The project team performs its activities

in accordance with its documented

contract tracking and oversight plans.

The project team performs its activities in

accordance with its documented contract

tracking and oversight plan.

Strength

Activity performed 2 The project team reviews required

contractor software planning

documents which, when satisfactory,

are used to oversee the contractor

team’s software effort.

The contractor did not prepare software

planning documents that the project team

could use to oversee the contractor’s

software development effort.

Weakness

Activity performed 3 The project team conducts periodic

reviews and interchanges with the

contractor team.

The project team does have periodic

reviews and discussions with the

contractor team that are documented.

Strength

Activity performed 4 The actual cost and schedule of the

contractor’s software effort are

compared to planned budgets and

schedules, and issues are identified.

The project team does compare the actual

cost and schedule of the contractor’s

software engineering effort to planned

schedules and budgets, identify issues,

and document results.

Strength

Activity performed 5 The size, critical computer resources,

and technical activities associated with

the contractor team’s work products

are tracked, and issues are identified.

The project team does track the size,

critical computer resources, and technical

activities associated with the contractor

team’s work products and identifies and

documents issues.

Strength
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 49 GAO-01-962 HUD Information Systems

Resident Assessment Subsystem

Common feature Key practice Finding Rating

Activity performed 6 The project team reviews and tracks

the development of the software

engineering environment required to

provide life-cycle support for the

acquired software, and issues are

identified.

The project team does review and track the

development of the software engineering

environment required to provide life-cycle

support for the acquired software, and

issues are identified and documented.

Strength

Activity performed 7 Any issues found by the project team

during contract tracking and oversight

activities are recorded in the

appropriate corrective action system,

action is taken, and issues are tracked

to closure.

The project team records issues related to

contract tracking and oversight in a

corrective action system, and tracks

corrective actions to closure.

Strength

Activity performed 8 The project team ensures that

changes to the software-related

contractual requirements are

coordinated with affected groups and

individuals, such as the contracting

official, contractor, and end user.

The project team stated that no changes to

the software-related contractual

requirements have taken place.

Not rated

Measurement and analysis 1 Measurements are made and used to

determine the status of the contract

tracking and oversight activities and

resultant products.

No measurements are made of HUD’s

contract tracking and oversight activities

and resultant products.

Weakness

Verifying implementation 1 Contract tracking and oversight

activities are reviewed by acquisition

management on a periodic basis.

The acquisition organization does not

review contract tracking and oversight

activities on a periodic basis.

Weakness

Verifying implementation 2 Contract tracking and oversight

activities are reviewed by the project

manager on both a periodic and an

event-driven basis.

The project manager informally reviews the

work of the project team. However, the

results of these reviews are not

documented.

Weakness
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 50 GAO-01-962 HUD Information Systems

Table 16: Contract Tracking and Oversight Findings for HUD’s Central Accounting and Program System

HUD’s Central Accounting and Program System

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for managing the

contract tracking and oversight of the

software acquisition project.

HUD has no written policy for contract

tracking and oversight of the acquisition of

software products and services. The

project team is following the policy in

HUD’s Procurement Handbook for contract

tracking and oversight.

Strength

Commitment to perform 2 Responsibility has been designated for

contract tracking and oversight

activities.

Responsibility for contract tracking and

oversight activities is designated in the

organization chart.

Strength

Commitment to perform 3 The project team includes contracting

specialists in the management of the

contract.

The project team includes contract

specialists to help manage the contract.

Strength

Ability to perform 1 A group is responsible for managing

contract tracking and oversight

activities.

A group is assigned responsibility for

contract tracking and oversight activities.

Strength

Ability to perform 2 Adequate resources are provided for

contract tracking and oversight

activities.

There is no mechanism to identify resource

needs and ensure that they are provided to

the project. The project manager stated

that the team does not have adequate

resources for performing contract tracking

and oversight activities.

Weakness

Ability to perform 3 Individuals performing contract

tracking and oversight activities have

experience or receive training.

Project team members performing contract

tracking and oversight activities have

experience and have received training.

Strength

Activity performed 1 The project team performs its activities

in accordance with its documented

contract tracking and oversight plans.

There is no contract tracking and oversight

plan; therefore, the activities are not

performed in accordance with it.

Weakness

Activity performed 2 The project team reviews required

contractor software planning

documents which, when satisfactory,

are used to oversee the contractor

team’s software effort.

The contractor did not prepare software

planning documents that the project team

could use to oversee the contractor’s

software development effort.

Weakness

Activity performed 3 The project team conducts periodic

reviews and interchanges with the

contractor team.

The project team does have periodic

reviews and discussions with the

contractor team that are documented.

Strength

Activity performed 4 The actual cost and schedule of the

contractor’s software effort are

compared to planned budgets and

schedules, and issues are identified.

The project team told us they do compare

the actual cost and schedule of the

contractor’s software engineering effort to

planned schedules and budgets, but no

documentation was provided to support

this statement.

Observation

Activity performed 5 The size, critical computer resources,

and technical activities associated with

the contractor team’s work products

are tracked, and issues are identified.

The project team does track the size,

critical computer resources, and technical

activities associated with the contractor

team’s work products and identifies and

documents issues.

Strength
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 51 GAO-01-962 HUD Information Systems

HUD’s Central Accounting and Program System

Common feature Key practice Finding Rating

Activity performed 6 The project team reviews and tracks

the development of the software

engineering environment required to

provide life-cycle support for the

acquired software, and issues are

identified.

The project team does review and track the

development of the software engineering

environment required to provide life-cycle

support for the acquired software, and

issues are identified and documented.

Strength

Activity performed 7 Any issues found by the project team

during contract tracking and oversight

activities are recorded in the

appropriate corrective action system,

action is taken, and issues are tracked

to closure.

The project team does not have a

corrective action system to record or track

action taken on issues found by the project

team during contract tracking and oversight

activities.

Weakness

Activity performed 8 The project team ensures that

changes to the software-related

contractual requirements are

coordinated with affected groups and

individuals, such as the contracting

official, contractor, and end user.

The project team stated that no changes to

the software-related contractual

requirements have taken place because

their contracts are broad enough to cover

changes to contractual requirements.

Not rated

Measurement and analysis 1 Measurements are made and used to

determine the status of the contract

tracking and oversight activities and

resultant products.

No measurements are made of HUD’s

contract tracking and oversight activities or

resultant products.

Weakness

Verifying implementation 1 Contract tracking and oversight

activities are reviewed by acquisition

management on a periodic basis.

The acquisition organization regularly

reviews contract tracking and oversight

activities.

Strength

Verifying implementation 2 Contract tracking and oversight

activities are reviewed by the project

manager on both a periodic and an

event-driven basis.

The project manager is briefed weekly on

project status, including contract tracking

and oversight activities.

Strength
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 52 GAO-01-962 HUD Information Systems

Table 17: Contract Tracking and Oversight Findings for the Empowerment Information System

Empowerment Information System

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for managing the

contract tracking and oversight of the

software acquisition project.

HUD has no written policy for contract

tracking and oversight of the acquisition of

software products and services. The

project team is following the policy in

HUD’s Procurement Handbook for

contract tracking and oversight.

Strength

Commitment to perform 2 Responsibility has been designated for

contract tracking and oversight

activities.

Responsibility for contract tracking and

oversight activities has been designated

in the organization chart.

Strength

Commitment to perform 3 The project team includes contracting

specialists in the management of the

contract.

The project team includes contract

specialists to help manage the contract.

Strength

Ability to perform 1 A group is responsible for managing

contract tracking and oversight

activities.

A group is assigned responsibility for

managing contract tracking and oversight

activities.

Strength

Ability to perform 2 Adequate resources are provided for

contract tracking and oversight

activities.

There is no mechanism to identify

resource needs and ensure that they are

provided to the project. The project

manager indicated that the team does not

have adequate resources for performing

contract tracking and oversight activities.

Weakness

Ability to perform 3 Individuals performing contract

tracking and oversight activities have

experience or receive training.

Project team members performing

contract tracking and oversight activities

have experience and have received

training.

Strength

Activity performed 1 The project team performs its activities

in accordance with its documented

contract tracking and oversight plans.

There is no contract tracking and

oversight plan; therefore, the activities are

not performed in accordance with it.

Weakness

Activity performed 2 The project team reviews required

contractor software planning

documents which, when satisfactory,

are used to oversee the contractor

team’s software effort.

The contractor did not prepare software

planning documents that the project team

could use to oversee the contractor’s

software development effort.

Weakness

Activity performed 3 The project team conducts periodic

reviews and interchanges with the

contractor team.

The project team does have periodic

reviews and discussions with the

contractor team that are documented.

Strength

Activity performed 4 The actual cost and schedule of the

contractor’s software effort are

compared to planned budgets and

schedules, and issues are identified.

The project team does compare the

actual cost and schedule of the

contractor’s software engineering effort to

planned schedules and budgets, identify

issues, and document results.

Strength

Activity performed 5 The size, critical computer resources,

and technical activities associated with

the contractor team’s work products

are tracked, and issues are identified.

The project team reviewed the size,

critical computer resources, and technical

activities associated with the contractor

team’s work products only during prerelease testing.

Observation
Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable

Page 53 GAO-01-962 HUD Information Systems

Empowerment Information System

Common feature Key practice Finding Rating

Activity performed 6 The project team reviews and tracks

the development of the software

engineering environment required to

provide life-cycle support for the

acquired software, and issues are

identified.

The project does review and track

development of the software engineering

environment required to provide life-cycle

support for the acquired software, and

issues are identified and documented.

Strength

Activity performed 7 Any issues found by the project team

during contract tracking and oversight

activities are recorded in the

appropriate corrective action system,

action is taken, and issues are tracked

to closure.

The project team records issues related to

contract tracking and oversight in a

corrective action system, and tracks

corrective actions to closure.

Strength

Activity performed 8 The project team ensures that

changes to the software-related

contractual requirements are

coordinated with affected groups and

individuals, such as the contracting

official, contractor, and end user.

The project team stated that changes to

software-related contractual requirements

were coordinated with all affected groups

and individuals, but they provided no

documentation to support this statement.

Weakness

Measurement and analysis 1 Measurements are made and used to

determine the status of the contract

tracking and oversight activities and

resultant products.

No measurements are made of HUD’s

contract tracking and oversight activities

or resultant products.

Weakness

Verifying implementation 1 Contract tracking and oversight

activities are reviewed by acquisition

management on a periodic basis.

The acquisition organization periodically

reviews contract tracking and oversight

activities.

Strength

Verifying implementation 2 Contract tracking and oversight

activities are reviewed by the project

manager on both a periodic and an

event-driven basis.

The project manager works closely with

and informally reviews the work of the

project team. However, the results of

these reviews are not documented.

Observation
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 54 GAO-01-962 HUD Information Systems

HUD did not satisfy the criteria for the repeatable level of maturity in this

key process area because of the number and severity of key practice

weaknesses found. These weaknesses result in increased risk that HUD’s

software acquisition projects will not be evaluated effectively, software

products will not work effectively or meet mission needs, and cost,

schedule, and other performance goals will not be met.

The purpose of evaluation is to determine that the acquired software

products and services satisfy contract requirements before acceptance.

Within this key process area, the SA-CMM specifies key practices for each

of the five common features that an organization must implement

effectively to achieve the repeatable level of maturity. Generally, these

practices include having a written organizational policy for evaluating

acquired software products and services, documenting plans for

evaluating software products and services, developing and managing

evaluation requirements in conjunction with software technical

requirements, tracking contractor evaluation activities for compliance

with the contract, and measuring and reporting on the status and results of

evaluation activities to management.

For the 75 key practices in this key process area, HUD had 37 strengths, 33

weaknesses, and five observations. For the

• Commitment to perform feature, there were four strengths, four

weaknesses, and two observations.

• Ability to perform feature, there were 11 strengths and nine weaknesses.

• Activities performed feature, there were 19 strengths, nine weaknesses,

and two observations.

• Measurement and analysis feature, there were no strengths, four

weaknesses, and one observation.

• Verifying implementation feature, there were three strengths and seven

weaknesses.

In addition, none of the projects had strengths in all the key practices.

As a result of these weaknesses, HUD is exposed to increased risk that

software acquisition projects will not be evaluated effectively, that

acquired software will not work efficiently or meet mission needs, and that

cost, schedule, and other performance goals will not be met. HUD

acknowledges the need to improve its software evaluation processes and

is committed to improving its software and system acquisition capability.

The department has prepared a statement of work to obtain contract

support to begin a software process improvement effort. This document

Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 55 GAO-01-962 HUD Information Systems

specifies that the contractor would review HUD software acquisition

processes, prioritize the key processes and practices to address, and

develop a plan to improve HUD’s acquisition processes.

Details of our evaluation are provided in the following figure and tables.

Figure 5 provides a comprehensive listing of the strengths, weaknesses,

and observations in the evaluation key process area. The specific findings

supporting the practice ratings cited in figure 5 are in tables 18 through 22.
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 56 GAO-01-962 HUD Information Systems

Figure 5: Software Evaluation Summary

Common Projects

features Key practices PIC REMS RASS HUDCAPS EIS

Commitment to

perform 1

The acquisition organization has a written policy for

managing the evaluation of acquired software products

and services.

Weakness Weakness Observation Observation Weakness

Commitment to

perform 2

Responsibility has been designated for evaluation

activities. Weakness Strength Strength Strength Strength

Ability to

perform 1

A group is responsible for planning, managing, and

performing evaluation activities. Weakness Strength Strength Strength Strength

Ability to

perform 2

Adequate resources are provided for evaluation

activities. Weakness Weakness Strength Weakness Weakness

Ability to

perform 3

Individuals performing evaluation activities have

experience or receive training. Strength Strength Strength Strength Strength

Ability to

perform 4

Members of the project team and groups supporting the

software acquisition receive orientation on the

objectives of the evaluation approach.

Weakness Weakness Weakness Weakness Strength

Activity

performed 1

The project team performs its activities in accordance

with its documented evaluation plans. Strength Strength Strength Weakness Weakness

Activity

performed 2

The project’s evaluation requirements are developed in

conjunction with the development of the system or

software technical requirements.

Strength Observation Strength Strength Weakness

Activity

performed 3

The project’s evaluation activities are planned to

minimize duplication and take advantage of all

evaluation results, where appropriate.

Strength Strength Strength Strength Weakness

Activity

performed 4

The project team appraises the contractor team’s

performance over the total period of the contract for

compliance with requirements.

Weakness Observation Strength Strength Weakness

Activity

performed 5

Planned evaluations are performed on the evolving

software products and services before acceptance for

operational use.

Strength Strength Strength Strength Weakness

Activity

performed 6

Results of the evaluations are analyzed and compared

to the contract’s requirements to establish an objective

basis to support the decision to accept the product or

services or to take further action.

Weakness Strength Strength Strength Weakness

Measurement

and analysis 1

Measurements are made and used to determine the

status of the evaluation activities and resultant products. Weakness Weakness Weakness Weakness Observation

Verifying

implementation 1

Evaluation activities are reviewed by acquisition

organization management on a periodic basis. Strength Strength Weakness Weakness Weakness

Verifying

implementation 2

Evaluation activities are reviewed by the project

manager on both a periodic and an event-driven basis. Weakness Strength Weakness Weakness Weakness

Key Strength Strength: Key practice effectively implemented.

Weakness Weakness: Key practice not effectively implemented.

Observation Observation: Key practice evaluated; evidence not sufficient to rate as a strength, or practice only partially performed.
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 57 GAO-01-962 HUD Information Systems

Table 18: Software Evaluation Findings for the Public and Indian Housing Information Center System

Public and Indian Housing Information Center system

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for managing the

evaluation of acquired software

products and services.

HUD has no written policy for managing the

evaluation of acquired software products

and services.

Weakness

Commitment to perform 2 Responsibility has been designated

for evaluation activities.

The project team told us that responsibility

for evaluation activities has been assigned

to the lead technical person, but no

documentation was provided to support this

statement.

Weakness

Ability to perform 1 A group is responsible for planning,

managing, and performing evaluation

activities.

The project team told us that evaluation

activities are managed by the lead technical

person, but no documentation was provided

to support this statement.

Weakness

Ability to perform 2 Adequate resources are provided for

evaluation activities.

There is no mechanism to identify needed

resources and ensure that they are provided

to the project. The lead technical person

indicated that the team does not have

adequate resources for performing

evaluation activities.

Weakness

Ability to perform 3 Individuals performing evaluation

activities have experience or receive

training.

Project team members performing

evaluation activities have experience.

Strength

Ability to perform 4 Members of the project team and

groups supporting the software

acquisition receive orientation on the

objectives of the evaluation

approach.

Members of the project team or others

supporting the software acquisition did not

receive orientation on the objectives of the

evaluation approach.

Weakness

Activity performed 1 The project team performs its

activities in accordance with its

documented evaluation plans.

The project team performs its evaluation

activities in accordance with its documented

evaluation plans.

Strength

Activity performed 2 The project’s evaluation requirements

are developed in conjunction with the

development of the system or

software technical requirements.

Project evaluation requirements were

developed in conjunction with software

technical requirements and are designated

in the test plans and test scripts.

Strength

Activity performed 3 The project’s evaluation activities are

planned to minimize duplication and

take advantage of all evaluation

results, where appropriate.

There appears to be no duplication of

evaluation activities.

Strength

Activity performed 4 The project team appraises the

contractor team’s performance over

the total period of the contract for

compliance with requirements.

The project team stated that quarterly

reports of contractor deliverables were used

to appraise contractor performance, but no

documentation was provided to support this

statement.

Weakness

Activity performed 5 Planned evaluations are performed

on the evolving software products

and services before acceptance for

operational use.

Several evaluations are performed on

evolving software products and services

before acceptance for operational use.

Strength
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 58 GAO-01-962 HUD Information Systems

Public and Indian Housing Information Center system

Common feature Key practice Finding Rating

Activity performed 6 Results of the evaluations are

analyzed and compared to the

contract’s requirements to establish

an objective basis to support the

decision to accept the product or

services or to take further action.

There is no analysis or comparison of

evaluation results to contract requirements

to establish an objective basis supporting

the decision to accept the products and

services or to take further action.

Weakness

Measurement and analysis 1 Measurements are made and used to

determine the status of the evaluation

activities and resultant products.

No measurements are made of HUD’s

evaluation activities and resultant products.

Weakness

Verifying implementation 1 Evaluation activities are reviewed by

acquisition organization management

on a periodic basis.

Acquisition organization management

regularly reviews evaluation activities.

Strength

Verifying implementation 2 Evaluation activities are reviewed by

the project manager on both a

periodic and an event-driven basis.

The project manager does not formally

review evaluation activities, and the project

manager’s reviews are not documented.

Weakness
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 59 GAO-01-962 HUD Information Systems

Table 19: Software Evaluation Findings for the Real Estate Management System

Real Estate Management System

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for managing the

evaluation of acquired software

products and services.

HUD has no written policy for managing the

evaluation of acquired software products

and services.

Weakness

Commitment to perform 2 Responsibility has been designated for

evaluation activities.

Contractors and users have been assigned

responsibility for evaluation activities.

Strength

Ability to perform 1 A group is responsible for planning,

managing, and performing evaluation

activities.

The contractor and end users are

responsible for evaluation activities.

Strength

Ability to perform 2 Adequate resources are provided for

evaluation activities.

There is no mechanism to identify needed

resources and ensure that they are provided

to the project. The project manager stated

that the team does not have adequate

resources for performing evaluation

activities.

Weakness

Ability to perform 3 Individuals performing evaluation

activities have experience or receive

training.

Project team members performing

evaluation activities have experience.

Strength

Ability to perform 4 Members of the project team and

groups supporting the software

acquisition receive orientation on the

objectives of the evaluation approach.

Members of the project team or others

supporting the software acquisition did not

receive orientation on the objectives of the

evaluation approach.

Weakness

Activity performed 1 The project team performs its activities

in accordance with its documented

evaluation plans.

The team performs its evaluation activities in

accordance with its documented evaluation

plans.

Strength

Activity performed 2 The project’s evaluation requirements

are developed in conjunction with the

development of the system or software

technical requirements.

Project evaluation requirements are

documented in the project plan, but no

documentation was provided showing that

they were developed in conjunction with

development of the software technical

requirements.

Observation

Activity performed 3 The project’s evaluation activities are

planned to minimize duplication and

take advantage of all evaluation results,

where appropriate.

There appears to be no duplication of

evaluation activities.

Strength

Activity performed 4 The project team appraises the

contractor team’s performance over the

total period of the contract for

compliance with requirements.

The project team assesses contractor team

performance for compliance with

requirements only during user acceptance

testing.

Observation

Activity performed 5 Planned evaluations are performed on

the evolving software products and

services before acceptance for

operational use.

Planned evaluations are performed before

software releases are accepted for

operational use.

Strength
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 60 GAO-01-962 HUD Information Systems

Real Estate Management System

Common feature Key practice Finding Rating

Activity performed 6 Results of the evaluations are analyzed

and compared to the contract’s

requirements to establish an objective

basis to support the decision to accept

the product or services or to take

further action.

The results of evaluations are analyzed and

compared to the contract’s requirements to

establish an objective basis supporting the

decision to accept the products and services

or to take further action.

Strength

Measurement and analysis 1 Measurements are made and used to

determine the status of the evaluation

activities and resultant products.

No measurements are made of HUD’s

evaluation activities or resultant products.

Weakness

Verifying implementation 1 Evaluation activities are reviewed by

acquisition organization management

on a periodic basis.

Acquisition organization management

regularly reviews evaluation activities.

Strength

Verifying implementation 2 Evaluation activities are reviewed by

the project manager on both a periodic

and an event-driven basis.

Project reviews conducted by the project

manager cover evaluation activities.

Strength
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 61 GAO-01-962 HUD Information Systems

Table 20: Software Evaluation Findings for the Resident Assessment Subsystem

Resident Assessment Subsystem

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for managing the

evaluation of acquired software

products and services.

HUD has no written policy for managing the

evaluation of acquired software products and

services. The project is following the

evaluation policies in HUD’s software

development methodology.

Observation

Commitment to perform 2 Responsibility has been designated

for evaluation activities.

Responsibility for evaluation activities has

been assigned to the IT project lead.

Strength

Ability to perform 1 A group is responsible for planning,

managing, and performing evaluation

activities.

The IT project lead and support staff are

responsible for evaluation activities.

Strength

Ability to perform 2 Adequate resources are provided for

evaluation activities.

There is no mechanism to identify resource

needs and ensure that they are provided to

the project. According to the project manager,

the project team has adequate resources for

conducting evaluation activities.

Strength

Ability to perform 3 Individuals performing evaluation

activities have experience or receive

training.

Project team members performing evaluation

activities have experience and have received

training.

Strength

Ability to perform 4 Members of the project team and

groups supporting the software

acquisition receive orientation on the

objectives of the evaluation

approach.

Members of the project team or others

supporting the software acquisition did not

receive orientation on the objectives of the

evaluation approach.

Weakness

Activity performed 1 The project team performs its

activities in accordance with its

documented evaluation plans.

The project team performs its evaluation

activities in accordance with its documented

evaluation plan.

Strength

Activity performed 2 The project’s evaluation requirements

are developed in conjunction with the

development of the system or

software technical requirements.

Project evaluation requirements are developed

in conjunction with software technical

requirements.

Strength

Activity performed 3 The project’s evaluation activities are

planned to minimize duplication and

take advantage of all evaluation

results, where appropriate.

There appears to be no duplication of

evaluation activities.

Strength

Activity performed 4 The project team appraises the

contractor team’s performance over

the total period of the contract for

compliance with requirements.

The project team appraises the contractor

team’s performance over the total period of

the contract for compliance with requirements.

Strength

Activity performed 5 Planned evaluations are performed

on the evolving software products

and services before acceptance for

operational use.

Planned evaluation activities are performed on

the evolving software products and services

before they are accepted for operational use.

Strength

Activity performed 6 Results of the evaluations are

analyzed and compared to the

contract’s requirements to establish

an objective basis to support the

decision to accept the product or

services or to take further action.

Results of the evaluations are analyzed and

compared to the contract’s requirements to

establish an objective basis supporting the

decision to accept the software products or to

take further action.

Strength
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 62 GAO-01-962 HUD Information Systems

Resident Assessment Subsystem

Common feature Key practice Finding Rating

Measurement and analysis 1 Measurements are made and used to

determine the status of the evaluation

activities and resultant products.

No measurements are made of HUD’s

evaluation activities and resultant products.

Weakness

Verifying implementation 1 Evaluation activities are reviewed by

acquisition organization management

on a periodic basis.

Acquisition organization management does

not review evaluation activities on a regular

basis.

Weakness

Verifying implementation 2 Evaluation activities are reviewed by

the project manager on both a

periodic and an event-driven basis.

The project manager stated that while he

meets with his staff daily to discuss the status

of the project, meeting results and action items

are not documented.

Weakness
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 63 GAO-01-962 HUD Information Systems

Table 21: Software Evaluation Findings for HUD’s Central Accounting and Program System

HUD’s Central Accounting and Program System

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for managing the

evaluation of acquired software

products and services.

HUD has no written policy for managing the

evaluation of acquired software products

and services. The project is following the

evaluation policy in HUD’s software

development methodology.

Observation

Commitment to perform 2 Responsibility has been designated for

evaluation activities.

Responsibility for evaluation activities has

been assigned to the project lead.

Strength

Ability to perform 1 A group is responsible for planning,

managing, and performing evaluation

activities.

The project lead and support staff are

responsible for evaluation activities.

Strength

Ability to perform 2 Adequate resources are provided for

evaluation activities.

There is no mechanism to identify needed

resources and ensure that they are provided

to the project. The project manager stated

that the team does not have adequate

resources for performing evaluation

activities.

Weakness

Ability to perform 3 Individuals performing evaluation

activities have experience or receive

training

Project team members performing

evaluation activities have experience and

have received training.

Strength

Ability to perform 4 Members of the project team and

groups supporting the software

acquisition receive orientation on the

objectives of the evaluation approach.

Members of the project team and groups

supporting the software acquisition did not

receive orientation on the objectives of the

evaluation approach.

Weakness

Activity performed 1 The project team performs its activities

in accordance with its documented

evaluation plans.

There is no evaluation plan; therefore, the

activities are not performed in accordance

with it.

Weakness

Activity performed 2 The project’s evaluation requirements

are developed in conjunction with the

development of the system or software

technical requirements.

Project evaluation requirements were

developed in conjunction with software

technical requirements and are designated

in the statement of work.

Strength

Activity performed 3 The project’s evaluation activities are

planned to minimize duplication and

take advantage of all evaluation results,

where appropriate.

There appears to be no duplication of

evaluation activities.

Strength

Activity performed 4 The project team appraises the

contractor team’s performance over the

total period of the contract for

compliance with requirements.

The project team appraises the contractor

team’s performance over the total period of

the contract for compliance with

requirements.

Strength

Activity performed 5 Planned evaluations are performed on

the evolving software products and

services before acceptance for

operational use.

Planned evaluation activities are performed

on the evolving software products and

services before they are accepted for

operational use.

Strength

Activity performed 6 Results of the evaluations are analyzed

and compared to the contract’s

requirements to establish an objective

basis to support the decision to accept

the product or services or to take

further action.

Results of the evaluations are analyzed and

compared to the contract’s requirements to

establish an objective basis to support the

decision to accept the software products or

to take further action.

Strength
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 64 GAO-01-962 HUD Information Systems

HUD’s Central Accounting and Program System

Common feature Key practice Finding Rating

Measurement and analysis 1 Measurements are made and used to

determine the status of the evaluation

activities and resultant products.

No measurements are made of HUD’s

evaluation activities or resultant products.

Weakness

Verifying implementation 1 Evaluation activities are reviewed by

acquisition organization management

on a periodic basis.

Acquisition organization management does

not review evaluation activities on a regular

basis.

Weakness

Verifying implementation 2 Evaluation activities are reviewed by

the project manager on both a periodic

and an event-driven basis.

The project manager stated that while he

discusses the status of the project with team

members, meeting results and action items

are not documented.

Weakness
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 65 GAO-01-962 HUD Information Systems

Table 22: Software Evaluation Findings for the Empowerment Information System

Empowerment Information System

Common feature Key practice Finding Rating

Commitment to perform 1 The acquisition organization has a

written policy for managing the

evaluation of acquired software

products and services.

HUD has no written policy for managing the

evaluation of acquired software products and

services.

Weakness

Commitment to perform 2 Responsibility has been designated for

evaluation activities.

Responsibility for evaluation activities has

been assigned to the project lead.

Strength

Ability to perform 1 A group is responsible for planning,

managing, and performing evaluation

activities.

The project lead and team members are

responsible for evaluation activities.

Strength

Ability to perform 2 Adequate resources are provided for

evaluation activities.

There is no mechanism to identify needed

resources and ensure that they are provided

to the project. The project manager stated

that the team does not have adequate

resources for performing evaluation activities.

Weakness

Ability to perform 3 Individuals performing evaluation

activities have experience or receive

training.

Project team members performing evaluation

activities have experience and have received

training.

Strength

Ability to perform 4 Members of the project team and

groups supporting the software

acquisition receive orientation on the

objectives of the evaluation approach.

Members of the project team or others

supporting the software acquisition did

receive orientation on the objectives of the

evaluation approach.

Strength

Activity performed 1 The project team performs its activities

in accordance with its documented

evaluation plans.

There is no evaluation plan; therefore, the

activities were not performed in accordance

with it.

Weakness

Activity performed 2 The project’s evaluation requirements

are developed in conjunction with the

development of the system or software

technical requirements.

The project’s evaluation requirements were

not developed in conjunction with the

development of the software technical

requirements.

Weakness

Activity performed 3 The project’s evaluation activities are

planned to minimize duplication and

take advantage of all evaluation

results, where appropriate.

There is no evidence of evaluation activities

being performed.

Weakness

Activity performed 4 The project team appraises the

contractor team’s performance over

the total period of the contract for

compliance with requirements.

The project team assessed contractor team

performance for compliance with

requirements only at the end of the effort.

Weakness

Activity performed 5 Planned evaluations are performed on

the evolving software products and

services before acceptance for

operational use.

No planned evaluations were performed on

the software. A user survey was conducted,

and the software was rejected on the basis of

the survey.

Weakness

Activity performed 6 Results of the evaluations are

analyzed and compared to the

contract’s requirements to establish an

objective basis to support the decision

to accept the product or services or to

take further action.

No planned evaluations were conducted. The

software was rejected based on the results of

the user survey.

Weakness
Chapter 5: HUD’s Software Evaluation

Processes Are Not Repeatable

Page 66 GAO-01-962 HUD Information Systems

Empowerment Information System

Common feature Key practice Finding Rating

Measurement and analysis 1 Measurements are made and used to

determine the status of the evaluation

activities and resultant products.

No measurements are made of HUD’s

evaluation activities or resultant products.

The cost of the user survey was reported.

Observation

Verifying implementation 1 Evaluation activities are reviewed by

acquisition organization management

on a periodic basis.

Acquisition organization management does

not review evaluation activities on a regular

basis.

Weakness

Verifying implementation 2 Evaluation activities are reviewed by

the project manager on both a periodic

and an event-driven basis.

The project manager does not formally

review evaluation activities, and the project

manager’s reviews are not documented.

Weakness
hapter 6: Conclusions, Recommendations, and

Agency Comments

Page 67 GAO-01-962 HUD Information Systems

HUD’s software acquisition processes are immature and are not repeatable

on a project-by-project basis because of the many weaknesses it has in the

key practices. Strong performance of the key practices is essential for

achieving effective, repeatable, and lasting implementation and

institutionalization of the four key process areas we reviewed. Currently,

HUD’s success or failure in acquiring software depends largely on specific

individuals, rather than on well-defined and disciplined software

acquisition management practices. As a result, HUD is exposed to higher

risks that software-intensive acquisition projects will not consistently meet

mission requirements, perform as intended, or be delivered on schedule

and within budget.

HUD acknowledged that it has software acquisition weaknesses and is

committed to improving its software and system acquisition processes. We

agree that HUD should begin a software process improvement effort.

However, for HUD to successfully complete such an effort, its software

process improvement plan must address several important issues. To be

comprehensive, this plan should include the results of our review and

include measurable goals and time frames, estimates of resource

requirements, and time frames to reach the repeatable level. It should also

contain steps to address the systemic weaknesses we found. Finally, the

plan should include steps to address the areas we did not review and to

ensure that all ongoing as well as new software acquisition projects adopt

processes and meet the requirements for the repeatable level of maturity.

To improve HUD’s software acquisition capabilities, GAO recommends

that the Secretary of Housing and Urban Development direct the HUD

Chief Information Officer to develop and implement a comprehensive plan

for software acquisition process improvement that is based on the

software capability results in this report and specifies measurable goals

and time frames, sets priorities for initiatives, estimates resource

requirements (for trained staff and funding), and defines a process

improvement management structure.

Also, to address the systemic weaknesses mentioned above, the plan

should contain steps to

• develop a comprehensive policy for the acquisition of software,

• require plans for specific acquisition activities,

• measure and track the status of activities performed by the software

acquisition teams, and

• report regularly to management on progress and problems.

hapter 6: Conclusions, Recommendations,

and Agency Comments
hapter 6: Conclusions, Recommendations, and

Agency Comments

Page 68 GAO-01-962 HUD Information Systems

In addition, to ensure that all aspects of software acquisition are

addressed, we recommend that the Secretary direct the CIO to

• assess HUD’s maturity in the three key process areas that could not be

evaluated by GAO and include any needed improvement actions in the

comprehensive plan for software acquisition process improvement;

• ensure that all new software acquisition projects in HUD adopt processes

that satisfy at least SA-CMM level 2 requirements; and

• ensure that process improvement activities are initiated for all ongoing

software acquisition projects.

In providing written comments on a draft of this report, the Chief

Information Officer (CIO) stated that HUD concurred with our

recommendations to strengthen its software acquisition processes. The

CIO acknowledged that its software acquisition processes have

weaknesses and stated that it has an improvement project under way. This

project includes plans to seek contractor assistance to address the

weaknesses we identified and help improve HUD’s software acquisition

processes. Finally, the CIO stated that the department generally agreed

with our assessment of its software acquisition practices. HUD also

provided additional documentation on its practices for the Resident

Assessment Subsystem. We reviewed this information and made revisions

where appropriate and supported by the additional documentation. The

department’s comments are reprinted as appendix I.

Agency Comments

and Our Evaluation
Appendix I: Comments From the Department of Housing and Urban Development

Page 69 GAO-01-962 HUD Information Systems

Appendix I: Comments From the Department

of Housing and Urban Development
Appendix I: Comments From the Department of Housing and Urban Development

Page 70 GAO-01-962 HUD Information Systems
Appendix II: GAO Contact and Staff Acknowledgments

Page 71 GAO-01-962 HUD Information Systems

David G. Gill (202) 512-6250

In addition to those named above, Naba Barkakati, Suzanne M. Burns,

John Christian, Barbara Collier, Tonia Johnson, Barbara Oliver, and

Madhav Panwar made key contributions to this report.

Appendix II: GAO Contact and Staff

Acknowledgments

GAO Contact

Staff

Acknowledgments

(310306)

The first copy of each GAO report is free. Additional copies of reports are

$2 each. A check or money order should be made out to the

Superintendent of Documents. VISA and MasterCard credit cards are also

accepted.

Orders for 100 or more copies to be mailed to a single address are

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

Orders by visiting:

Room 1100

700 4th St., NW (corner of 4th and G Sts. NW)

Washington, DC 20013

Orders by phone:

(202) 512-6000

fax: (202) 512-6061

TDD (202) 512-2537

Each day, GAO issues a list of newly available reports and testimony. To

receive facsimile copies of the daily list or any list from the past 30 days,

please call (202) 512-6000 using a touchtone phone. A recorded menu will

provide information on how to obtain these lists.

Orders by Internet

For information on how to access GAO reports on the Internet, send an email message with “info” in the body to

Info@www.gao.gov

or visit GAO’s World Wide Web home page at

http://www.gao.gov

Contact one:

• Web site: http://www.gao.gov/fraudnet/fraudnet.htm

• E-mail: fraudnet@gao.gov

• 1-800-424-5454 (automated answering system)

Ordering Information

To Report Fraud,

Waste, and Abuse in

Federal Programs