United States General Accounting Office

GAO Report to the Chairman, Subcommittee

on Social Security, Committee on Ways

and Means, House of Representatives

January 1998 SOCIAL SECURITY

ADMINISTRATION

Software Development

Process Improvements

Started But Work

Remains

GAO/AIMD-98-39

GAO United States

General Accounting Office

Washington, D.C. 20548

Accounting and Information

Management Division

B-276524 

January 28, 1998

The Honorable Jim Bunning

Chairman, Subcommittee on Social Security

Committee on Ways and Means

House of Representatives

Dear Mr. Chairman:

As you know, the Social Security Administration (SSA) is in the process of

redesigning its work processes and modernizing its computer systems in

order to better serve an increasing beneficiary population and achieve

improvements in productivity. A key aspect of this modernization program

is the agency’s transition from its current centralized mainframe-based

computer processing environment to a client/server processing

environment1

 in which its Intelligent Workstation/Local Area Network

(IWS/LAN)2

 will serve as the basic automation infrastructure.

Software developed for the new client/server systems will be critical to

ensuring that the modernized processes function as intended and achieve

the desired productivity outcomes. However, software development has

been identified by many experts as one of the riskiest and most costly

aspects of systems development. SSA has recognized weaknesses in its own

software development capability and has begun taking steps to improve

the processes and methods that it uses to develop software. As requested,

our objective was to review the status of SSA’s software development

process improvement efforts.

Results in Brief SSA has initiated a number of actions to improve its software development

capability. Among other things, it has (1) launched a formal software

process improvement program and initiated pilot projects to test improved

software development processes, (2) acquired the assistance of the

Software Engineering Institute (SEI)3

 to help it assess the strengths and

1

In a client/server environment, servers and individual workstations are all capable of performing tasks

that previously only the mainframe computer could accomplish. This can sometimes result in

improvements over mainframe performance.

2

In June 1996, SSA awarded a national IWS/LAN contract to modernize and standardize the distributed

processing environment in its headquarters and field components and in state Disability Determination

Services (DDS). Phase I of the IWS/LAN initiative will provide 56,000 workstations and 1,742 local area

networks to SSA and DDS offices nationwide.

3

SEI is a nationally recognized, federally funded research and development center established at

Carnegie Mellon University in Pittsburgh, Pennsylvania, to address software development issues.

Page 1 GAO/AIMD-98-39 Software Process Improvements
B-276524 

weaknesses in its current software development processes and to assist in

implementing the improvement program, and (3) established a

management steering committee and a software engineering process

group within the Office of Systems to oversee software process

improvement activities. These are positive steps that should help position

SSA to improve its software development capability.

Although these initiatives are underway, SSA has not yet established key

elements of its software process improvement program that are needed to

measure the progress and success of its improvement efforts. In particular,

SSA has not yet defined specific, measurable goals for its software process

improvement program or established the baseline data that it will use to

assess its progress in achieving these goals. Without this essential

information, SSA cannot be assured of the extent to which its improvement

efforts will result in the consistent and cost-effective production of

high-quality products.

Background SSA is currently undertaking a multiyear, multibillion dollar systems

modernization effort that is intended to replace aging equipment, support

current and future redesigned work processes, and improve productivity.

The cornerstone of this modernization effort is the agency’s transition

from its current centralized, mainframe-based computer processing

environment to a highly distributed client/server processing environment.

The IWS/LAN infrastructure—consisting of networks of intelligent

workstations connected to each other and to SSA’s mainframe

computers—is intended to provide SSA with the initial computing

framework for using client/server technology to achieve cost savings and

improve customer service by distributing selected processes and

information closer to where they are needed. Through fiscal year 1997, SSA

had reported spending approximately $565 million on acquiring

workstations, local area networks, and other services to support the

IWS/LAN infrastructure.4

Software development is a critical component of the modernization

initiative. SSA’s Office of Systems, with contractors’ assistance, is designing

and developing a new generation of software that is scheduled to operate

on the IWS/LAN to support redesigned work processes in a client/server

environment. It has selected the disability claims process as the first major

redesign effort and is currently developing the software—referred to as

4

The other services include site preparation, support services and training, and telecommunications

and maintenance.

Page 2 GAO/AIMD-98-39 Software Process Improvements
B-276524 

the Reengineered Disability System (RDS)—that is intended to automate

this redesigned process. RDS is scheduled for national implementation on

the IWS/LAN from July 1999 through May 2001.

To help SSA’s software development efforts succeed, however, it is

important that the agency have disciplined and consistent software

development practices that produce high-quality software within budget

and on schedule. In September 1996, we reported that SSA had experienced

problems in developing RDS, which contributed to a delay of more than 2

years in its scheduled implementation.5

 These problems included (1) using

programmers with insufficient experience, (2) using software development

tools that did not perform effectively, and (3) establishing initial software

development schedules that were too optimistic. During that same month,

a contractor’s preliminary risk assessment of SSA’s client/server transition

strategy identified various risks associated with the existing software

development processes, including ineffective requirements definition and

inadequate configuration management.

In addition, SSA is currently facing the critical challenge of ensuring that its

information systems are Year 2000 compliant. By the end of this century,

SSA must review all of its computer software and make the changes needed

to ensure that its systems can correctly process information relating to

dates. These changes affect not only its new network but computer

programs operating on both its mainframe and personal computers. We

recently reported that while SSA has made significant progress in its Year

2000 efforts, it faces the risk that not all of its mission-critical systems will

be corrected by the turn of the century.6

 At particular risk are the systems

used by state DDSs to help SSA process disability claims.

Making software process improvements to address problems such as

those SSA faces is considered to be a challenging undertaking for any

organization. To guide agencies in assessing the strengths and weaknesses

of their software development processes, SEI, in the late 1980s, developed

the Capability Maturity Model (CMM). CMM is organized into five levels that

characterize an organization’s software process maturity. As shown in

table 1, these levels range from initial (level 1), characterized by ad hoc

and chaotic processes, to optimizing (level 5), characterized by

5

Social Security Administration: Effective Leadership Needed to Meet Daunting Challenges

(GAO/AIMD-96-196, September 12, 1996).

6

Social Security Administration: Significant Progress Made in Year 2000 Effort, But Key Risks Remain

(GAO/AIMD-98-6, October 22, 1997).

Page 3 GAO/AIMD-98-39 Software Process Improvements
B-276524 

continuous process improvement based upon analysis and quantitative

data.

Table 1: Software Capability Maturity

Model Levels and Descriptions Level Name Description

5 Optimizing Continuous process improvement is enabled by

quantitative feedback from the process and from piloting

innovative ideas and technologies.

4 Managed Detailed measures of the software process and product

quality are collected. Both the software process and

products are quantitatively understood and controlled.

3 Defined The software process for both management and

engineering activities is documented, standardized, and

integrated into a standard software process for the

organization. All projects use an approved, tailored

version of the organization’s standard software process

for developing and maintaining software.

2 Repeatable Basic project management processes are established to

track cost, schedule, and functionality. The necessary

process discipline is in place to repeat earlier successes

on projects with similar applications.

1 Initial The software process is characterized as ad hoc and

occasionally even chaotic. Few processes are defined

and success depends on individual effort.

Source: Software Engineering Institute, Carnegie Mellon University, 1993.

Further, to assist agencies in implementing effective software process

improvement programs, SEI developed the IDEALSM model, which defines

five phases of process improvement activity.7

 These phases are:

Initiating. Management determines that there is a business reason for

improving their processes, sets general process improvement goals, and

sponsors a process improvement program.

Diagnosing. Using CMM, the current practices of the organization are

appraised and characterized. Results of the assessment are documented

and recommendations are made regarding areas in which to focus

improvement efforts.

Establishing. Based on the results of the diagnosing phase and the general

goals that were defined in the initiating phase, the organization develops a

strategy for improvement, prioritizes activities, and formulates measurable

7

IDEALSM: A User’s Guide for Software Process Improvement, SEI, February 1996.

Page 4 GAO/AIMD-98-39 Software Process Improvements
B-276524 

goals. Process action teams are established to develop action plans for

improvement.

Acting. Action plans are implemented through pilots. The results of pilots

are evaluated and action plans are modified as appropriate. When proven

effective, action plans are implemented throughout the organization.

Learning. After the new processes have been in place for some time, their

effectiveness is evaluated, communicated throughout the organization,

and, as appropriate, used to formulate new action plans to ensure that

goals are achieved.

Scope and

Methodology

To determine the status of SSA’s efforts to improve its software

development processes, we analyzed key documents, including SSA’s

Software Process Improvement Program Management Plan, dated

April 1997, Client/Server Transition Strategy: Preliminary Risk Assessment,

dated September 1996, and relevant systems and strategic planning

documents, such as the Information Systems Plan. In addition, to

determine the status of specific projects being undertaken by contractors

in support of the improvement initiatives, we reviewed the statements of

work for contractor services and final deliverables, such as baseline

assessments and software process improvement reports. We did not

independently verify the accuracy of information reported in the

contractors’ assessments of SSA’s software development processes.

We analyzed SEI’s IDEALSM: A User’s Guide for Software Process

Improvement, dated February 1996, which SSA is using to implement and

manage its process improvement program, to determine whether SSA’s

current and planned software development practices are consistent with

this guidance. We reviewed additional SEI reports, including Moving On

Up: Data and Experience Doing CMM-Based Process Improvement, dated

August 1995, to identify successful practices of organizations applying

CMM-based process improvements. In addition, we reviewed documents

discussing the implementation schedule, technical strategies, and risks

associated with SSA’s development of the RDS software application to

obtain information on the agency’s experiences in software development.

However, we did not specifically evaluate the progress of SSA’s ongoing

effort to develop RDS.

To further support our assessment of the actions that SSA is taking to

improve its software development capability, we interviewed the Deputy

Page 5 GAO/AIMD-98-39 Software Process Improvements
B-276524 

Commissioner for Systems and other systems officials directly involved in

implementing the improvement initiative, the General Services

Administration official responsible for administering the support contract

for SSA’s client/server software development assessment, and

representatives of the contractors involved in this initiative.

We performed our work from March 1997 through November 1997 in

accordance with generally accepted government auditing standards. SSA

provided written comments on a draft of this report. These comments are

discussed in the “Agency Comments and Our Evaluation” section and are

reprinted in appendix I.

SSA Has Initiated

Actions to Improve Its

Software

Development

Capability

Recognizing the need to improve its software development capability, SSA

has launched a formal software process improvement program and

initiated pilot projects to test improved software development processes.

In doing so, it is seeking to achieve a repeatable (level 2) level of software

capability maturity. SSA acquired the assistance of SEI to help formulate and

implement the improvement program. It adopted SEI’s CMM as the

framework for assessing the current state of its software development

capability and establishing improvement priorities and is following SEI’s

IDEALSM model as the methodology for implementing and managing its

software process improvement actions. Further, SSA has developed a

software process improvement schedule identifying the specific phases

and tasks that it plans to undertake to complete the implementation of its

improved software development processes by June 2000.

Consistent with SEI’s IDEALSM model, SSA has performed a number of the

steps recommended for the initiating, diagnosing, and establishing phases

of its software process improvement program and is beginning to

implement steps under the acting phase. It has put in place the initial

management infrastructure to support and facilitate its software process

improvement initiatives by establishing a management steering committee

to raise organizational awareness of the improvement program and a

software engineering process group to oversee the development and

implementation of the software process activities. It has also established

various work groups to carry out the specific process improvement

initiatives.8

8

The management steering committee is comprised primarily of senior managers, including the deputy

and assistant deputy commissioners for systems and associate commissioners within the Office of

Systems; the software engineering process group is comprised primarily of division director-level

representatives within the Office of Systems; and technical working groups include line staff from the

various systems components.

Page 6 GAO/AIMD-98-39 Software Process Improvements
B-276524 

SSA also has undertaken two assessments of the maturity of its existing

software development processes, focusing on identifying (1) effective

software development policies and procedures already being used within

the agency and (2) key software development process areas needing

improvement. SEI’s CMM specifies key process areas and criteria that must

be addressed to achieve a particular software development maturity level.

The key process areas for level 2 (repeatable) maturity—the level that SSA

is currently seeking to achieve—are (1) requirements management,

(2) software project planning, (3) software project tracking and oversight,

(4) software subcontract management, (5) software quality assurance, and

(6) software configuration management.

With SEI’s assistance, SSA conducted a self-assessment to determine the

strengths and weaknesses of its current processes for developing and

maintaining software, which are primarily mainframe-oriented. This

self-assessment identified 22 weaknesses in the level 2 key process areas.

For example, the assessment found within the area of software project

planning that risks had not been identified, assessed, or documented for

some projects, and within the area of software project tracking and

oversight, that the results and performance of some projects had not been

tracked against key elements—such as costs, schedules, and risks—of

SSA’s software project plans.

In addition, a support contractor hired to identify client/server best

practices and assist SSA in transitioning to a client/server environment

performed an independent assessment of the agency’s software

development processes. This assessment focused specifically on

identifying the strengths and weaknesses in SSA’s ability to develop

client/server software. It, too, identified weaknesses in SSA’s software

development practices. For example, the assessment identified within the

area of requirements management a need to improve practices for defining

system requirements and specifications, and in the area of project

planning, a need to improve practices for scheduling and estimating the

cost of software development efforts.

Based on the findings identified in both assessments, SSA developed a

software process improvement program implementation plan. This plan

will be used during the acting phase to, among other things, direct the

software development activities of three pilot projects that the Office of

Systems intends to undertake to help institutionalize the software process

Page 7 GAO/AIMD-98-39 Software Process Improvements
B-276524 

improvements.9 According to the implementation plan, project teams for

the three pilots are expected to test and evaluate improved software

development processes to address the identified weaknesses. SSA initiated

its pilot activities in September 1997 and, at the conclusion of our review,

had begun developing the policies and procedures that it will use to test

each of the key process areas. SSA expects to complete all of the pilots by

March 1999, after which it will finalize its strategy for implementing the

improved software development processes.

Software Process

Improvement

Program Lacks

Measurable Goals

Although SSA has made important progress in its efforts to improve its

software development processes, its improvement program does not yet

include specific, measurable goals and baseline data that are essential to

helping it achieve a repeatable (level 2) software development capability.

Without measurable goals and baseline data, SSA does not yet have critical

information needed to guide its improvement efforts and to provide

evidence that the efforts are resulting in more consistent, cost-effective,

and timely production of higher quality products.

According to SEI’s IDEALSM: A User’s Guide for Software Process

Improvement, clearly defined and measurable goals are necessary to

provide guidance and to assist in developing tactics for improving the

software development process. They also allow for objective measurement

of the improvement results. SEI prescribes that general goals of the

improvement program be defined during the initiating phase based on the

business needs of the organization. These general goals are used, in

conjunction with baseline data on the agency’s existing processes (such as

software size estimates, defects identified, and calendar time for project

completion), to develop specific short- and long-term measurable goals

during the establishing phase. For example, one general goal could be to

make software projects more predictable in terms of cost and schedule. If

the measurement baseline established that 80 percent of the organization’s

current projects exceed their original cost and schedule estimates by more

than 25 percent, then the specific, measurable goal could be to improve

that measure such that 80 percent of all projects are completed within

10 percent of their original cost and schedule estimates within 2 years.

9

The three projects that SSA plans to pilot are (1) the Customer Help and Information Program, a

workstation-based, client/server system that is to assist SSA in providing customers with consistent,

accurate responses to telephone requests; (2) the Consolidated Development Worksheet, a project in

which software will be developed to consolidate information, such as diaries, alerts, and actions items,

that is used by various SSA systems in processing claims, post-entitlement, and other workloads; and

(3) the Earnings Management Information Operational Data Store that in its first release is to provide

an integrated suite of applications for accessing and updating earnings information.

Page 8 GAO/AIMD-98-39 Software Process Improvements
B-276524 

At the conclusion of our review, SSA had established general goals for its

improvement program that included (1) achieving a repeatable (level

2) software capability maturity and (2) creating a software development

environment that encourages continuous improvements in quality,

productivity, and timeliness. However, it had not yet established specific,

measurable improvement goals for its software development processes,

nor had it defined the actual baseline data that it will use to monitor

progress in achieving its goals.

SSA’s Deputy Commissioner for Systems and the director of the software

process improvement program told us that they recognized the importance

of and need for establishing specific, measurable goals for the agency’s

improvement program. However, they stated that the agency has not yet

been able to define such goals for the improvement initiatives because it

has not traditionally maintained baseline data on its software development

projects that would be required to make such determinations. These

officials told us that they intend to develop the necessary measures based

on data compiled during the pilots being conducted to test improved

software development processes.

While SSA officials stated that they intend to develop the necessary

measures during the pilots, at the time of our review they did not have a

detailed strategy explaining how the pilots will be used to generate these

measures. SSA contracted with the Gartner Group to develop a strategic

measurement plan which recommends a general framework and steps for

establishing consistent, repeatable processes to collect and track

measurements; it also has outlined within its implementation plan the

general strategy it intends to use for the three pilots. However, neither of

these documents provided detailed information on how and in what time

frames baseline data and specific, measurable goals will be derived from

the pilots. Without an explicit strategy and time frames for generating

baseline data and measurable goals, SSA cannot ensure that it will have

essential information to monitor the progress of its improvement efforts.

Conclusions Recognizing the need to reassess its software development processes in

light of transitioning to a client/server processing environment, SSA is

taking important steps to improve its software development capability. If

successfully completed, these actions should help position the agency to

strengthen its processes for developing quality software products.

However, until SSA develops baseline data and establishes specific

measurable goals for its improvement initiatives as part of its pilot

Page 9 GAO/AIMD-98-39 Software Process Improvements
B-276524 

projects, it will not have necessary information to monitor its progress

toward achieving its intended software process improvements.

Recommendation To strengthen SSA’s software process improvement program, we

recommend that, as part of its recently initiated pilot projects, the

Commissioner of Social Security direct the Deputy Commissioner for

Systems to develop and implement plans that explicitly articulate SSA’s

strategy and time frames for (1) developing baseline data, (2) identifying

specific, measurable goals for its improvement initiative, and

(3) monitoring and measuring progress in achieving these goals.

Agency Comments

and Our Evaluation

In commenting on a draft of this report, SSA agreed with our

recommendation and described actions that it is undertaking to develop a

plan for its measurement activities. These actions include obtaining

support for its pilot projects from the Gartner Group and working with SEI

to formulate a plan that will include (1) tasks and time frames for

developing baseline data, (2) measurable goals for the implementation of

CMM-compliant processes, and (3) methods for measuring progress against

established goals.

We are encouraged by SSA’s response and will continue to monitor the

agency’s progress in implementing its software improvement effort.

As agreed with your office, unless you publicly announce the contents of

this report earlier, we plan no further distribution until 30 days from its

date. At that time, we will send copies to the Commissioner of Social

Security; the Director of the Office of Management and Budget;

appropriate congressional committees; and other interested parties.

Copies will also be made available to others upon request.

Page 10 GAO/AIMD-98-39 Software Process Improvements
B-276524 

Please contact me at (202) 512-6253 or by e:mail at

willemssenj.aimd@gao.gov if you have any questions concerning this

report. Major contributors to this report are listed in appendix II.

Sincerely yours,

Joel C. Willemssen

Director, Civil Agencies Information Systems

Page 11 GAO/AIMD-98-39 Software Process Improvements
Contents

Letter 1

Appendix I 

Comments From the

Social Security

Administration

14

Appendix II 

Major Contributors to

This Report

16

Table Table 1: Software Capability Maturity Model Levels and

Descriptions

4

Abbreviations

CMM Capability Maturity Model

DDS Disability Determination Service

IWS/LAN Intelligent Workstation/Local Area Network

RDS Reengineered Disability System

SEI Software Engineering Institute

SSA Social Security Administration

Page 12 GAO/AIMD-98-39 Software Process Improvements
Page 13 GAO/AIMD-98-39 Software Process Improvements
Appendix I 

Comments From the Social Security

Administration

Page 14 GAO/AIMD-98-39 Software Process Improvements
Appendix I 

Comments From the Social Security

Administration

Page 15 GAO/AIMD-98-39 Software Process Improvements
Appendix II 

Major Contributors to This Report

Accounting and

Information

Management Division,

Washington, D.C.

Valerie C. Melvin, Assistant Director

Leonard J. Latham, Technical Assistant Director

Michael A. Alexander, Senior Information Systems Analyst

Gwendolyn M. Adelekun, Business Process Analyst

(511216) Page 16 GAO/AIMD-98-39 Software Process Improvements
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100