a

GAO

United States General Accounting Office

Report to the Chairman, Subcommittee 

on Social Security, Committee on Ways 

and Means, House of Representatives

March 2004 ELECTRONIC 

DISABILITY CLAIMS 

PROCESSING

SSA Needs to Address 

Risks Associated with 

Its Accelerated 

Systems Development 

Strategy

GAO-04-466
www.gao.gov/cgi-bin/getrpt?GAO-04-466.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Linda D. 

Koontz at (202) 512-6240 or 

koontzl@gao.gov. 

Highlights of GAO-04-466, a report to the 

Chairman, Subcommittee on Social 

Security, Committee on Ways and Means, 

House of Representatives 

March 2004

ELECTRONIC DISABILITY CLAIMS 

PROCESSING 

SSA Needs to Address Risks Associated 

with Its Accelerated Systems 

Development Strategy 

SSA is continuing its work on the AeDib initiative and is in various stages of 

completing its electronic disability system; however, its accelerated strategy 

continues to involve risks. Specifically, GAO found that the agency is relying 

on limited pilot testing to help guide business and technical decisions and 

ensure that technology supporting the electronic disability system will work 

as intended (see table). Further, it is beginning its national rollout without 

ensuring that all critical problems identified in the pilot testing have been 

resolved and without conducting testing adequate to evaluate the 

performance of all system components collectively. Without resolution of 

critical problems and full testing, SSA cannot be assured that interrelated 

components will work together successfully. 

Pilot Tests are Limited in Scope, Involving Few Examiners 

State/ 

pilot start date 

Number/percentage of 

examiners participating 

North Carolina/ 

July 2003 

21 of 254/ 

8 percent 

Illinois/ 

September 2003 

15 of 250/ 

6 percent 

California/ 

October 2003 

2 of 620/ 

0.3 percent 

Source: GAO analysis of SSA data. 

While SSA has procedures to guide its software development, it could 

provide no evidence that it was consistently applying them in this case. In 

addition, while SSA has identified AeDib system and security risks, it has not 

finalized mitigation strategies. Without these measures being in place, SSA 

stands at greater vulnerability to circumstances that could impede project 

success. 

The agency also has not validated its analysis to ensure the reasonableness 

of estimated AeDib costs and benefits. While indicating that it would use 

pilot test results to validate cost-benefit estimates, officials have not 

indicated when this will be accomplished. This leaves SSA without a 

validated cost-benefit analysis, and the assurance that its AeDib cost 

estimates are reliable and that anticipated benefits will therefore be realized.

Finally, SSA reports that it has increased its communications with AeDib 

stakeholders and users; however, state officials dealing with disability 

determinations have varying perspectives. A national organization 

representing these state officials continues to voice concerns about SSA’s 

approach. And while the Commissioner states that SSA is consulting with 

stakeholders and the medical community, the agency has not articulated a 

comprehensive plan for ensuring that the concerns of this population are 

addressed.

The Social Security 

Administration’s (SSA) AeDib 

initiative is designed to provide 

SSA with a more efficient, 

paperless system that will enable 

its disability components to 

electronically view and share 

claims data and process claims 

electronically. Yet previous GAO 

reviews found that SSA’s 

accelerated strategy to develop 

AeDib involved risks that could 

threaten a complete and successful 

transition to this capability. 

At the Subcommittee’s request, 

GAO reviewed AeDib to assess (1) 

SSA’s progress and strategy, (2) the 

adequacy of measures taken to 

avoid software development 

problems similar to those 

encountered in SSA’s previous 

efforts, (3) the adequacy of costbenefit analyses, and (4) SSA’s 

consultation with stakeholders. 

GAO recommends that before 

proceeding with national rollout of 

AeDib, the Commissioner of Social 

Security, among other steps, ensure 

that critical problems have been 

resolved and full testing completed, 

expedite completion of riskmitigation strategies, and validate 

cost-benefit estimates. In 

commenting on a draft of this 

report, SSA stated that it would 

conduct studies to help validate 

AeDib cost assumptions but 

disagreed with GAO’s other 

recommendations. In GAO’s view, 

it is essential that SSA fully address 

all recommendations. 
Page i GAO-04-466 SSA’s AeDib Strategy

Contents

Letter 1

Recommendations for Executive Action 3

Agency Comments and Our Evaluation 4

Appendixes

Appendix I: GAO’s January 30, 2004, Briefing 8

Appendix II: Comments from the Social Security Administration 52

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately.
Page 1 GAO-04-466 SSA’s AeDib Strategy

United States General Accounting Office

Washington, D.C. 20548

A

March 26, 2004 Letr

The Honorable E. Clay Shaw, Jr.

Chairman, Subcommittee on Social Security

Committee on Ways and Means

House of Representatives

Dear Mr. Chairman:

As you know, the Social Security Administration (SSA) is currently 

pursuing a major initiative to establish an electronic disability claims 

processing capability. With this initiative—known as AeDib—SSA aims to 

develop and implement a more efficient, paperless system that will enable 

its disability processing components to electronically view and share 

claims information, including large volumes of medical images, files, and 

other documents currently maintained in paper folders. SSA’s pursuit of an 

electronic disability process represents a positive and necessary step 

toward more efficient delivery of benefits payments and services to an 

increasing beneficiary population. However, as we have previously 

reported,1

 the agency’s accelerated strategy to develop the electronic 

disability system involves risks that could pose a threat to the complete and 

successful transition to this capability.

At your request, we have been reviewing SSA’s continued actions on the 

AeDib initiative. Specifically, our objectives were to 

• assess SSA’s progress toward and strategy for achieving AeDib,

• determine the adequacy of measures SSA is taking with AeDib to avoid 

the kinds of software development problems that it encountered with its 

prior disability system initiative and to identify and mitigate risks to a 

successful development,

• assess the adequacy of SSA’s analysis of AeDib costs and benefits, and 

• assess SSA’s consultation with and support from key stakeholders.

1

U.S. General Accounting Office, Electronic Disability Claims Processing: Social Security 

Administration’s Accelerated Strategy Faces Significant Risks, GAO-03-984T (Washington, 

D.C.: July 24, 2003) and Social Security Administration: Subcommittee Questions 

Concerning Efforts to Automate the Disability Claims Process, GAO-03-1113R

(Washington, D.C.: Sept. 5, 2003).
Page 2 GAO-04-466 SSA’s AeDib Strategy

To address these objectives, we analyzed project management and 

technical documentation describing SSA’s plans, strategies, and progress 

related to developing and implementing the electronic disability system. 

We identified and evaluated measures—including institutional software 

capability mechanisms—that the agency had taken to guide its 

development of the AeDib software and mitigate program and project risks. 

We also assessed the adequacy of SSA’s analyses of AeDib costs and 

benefits, including evaluating them against federal guidance and industry 

best practices and analyzing the impact of critical pilot evaluation data on 

costs and benefits. In addition, we analyzed SSA’s plans and actions for 

consulting with and identifying and resolving issues of key disability 

stakeholders (i.e., state Disability Determination Services [DDS] offices, 

medical providers, and SSA field offices). We performed our work in 

accordance with generally accepted government auditing standards, from 

August 2003 through January 2004.

On January 30, 2004, we provided your staff with a briefing on the results of 

our study. The slides from that briefing are included as appendix I to this 

report. The purpose of this report is to provide the published briefing slides 

to you and to officially transmit our recommendations to the Commissioner 

of Social Security.

In summary, our briefing made four main points:

• While the AeDib initiative is important to achieving more efficient 

delivery of disability payments, SSA’s accelerated strategy continues to 

involve risks that threaten its ultimate success. SSA is relying on limited 

pilot testing to help guide business and technical decisions and ensure 

that technology supporting the electronic disability folder will work as 

intended. Further, the agency is beginning its national rollout without 

ensuring that all critical problems identified during the pilot tests have 

been resolved and without conducting end-to-end tests to evaluate the 

functionality and performance of all electronic disability system 

components collectively. SSA has maintained that its pilot tests of the 

components would be sufficient to evaluate the system; however, 

without end-to-end testing, SSA lacks assurance that the interrelated 

components will work together successfully.

• While SSA has established processes and procedures to guide its 

software development activities, such as project management plans, it 

could not provide evidence that it was consistently applying these 

procedures to AeDib. In addition, while SSA has identified AeDib system 
Page 3 GAO-04-466 SSA’s AeDib Strategy

and security risks, it has not finalized mitigation strategies. Without 

concurrence on software validation and system certifications, or risk 

mitigation strategies, SSA lacks assurance that its system will be 

acceptable to end users and it will not be positioned to effectively 

prevent circumstances that could impede project success.

• SSA has not validated its analysis to ensure the reasonableness of 

estimated AeDib costs and benefits. The agency stated that it would use 

pilot test results to validate its cost-benefit estimates; however, its 

officials have not indicated when this will be accomplished. Without a 

validated cost-benefit analysis, SSA lacks assurance that its AeDib cost 

estimates are reliable and that anticipated benefits will be realized.

• While SSA has stated that it has increased its communications with 

AeDib stakeholders and users and instituted mechanisms to facilitate its 

communications, state DDSs have varying views regarding their 

involvement. Specifically, while state disability offices participating in 

the AeDib pilots agree that communications have improved, a national 

organization representing these offices continues to have concerns 

about SSA’s approach. In addition, although the Commissioner has 

stated that SSA is consulting with stakeholders and the medical 

community, SSA has not articulated a comprehensive plan for ensuring 

that all of these concerns are addressed.

In light of the risks associated with the AeDib initiative, SSA’s successful 

transition to an electronic disability claims process could be jeopardized 

unless it addresses the weaknesses described above. 

Recommendations for 

Executive Action

To reduce the risks associated with SSA’s strategy to develop an electronic 

disability claims processing system, we recommend that the Commissioner 

of Social Security, before continuing with the AeDib national rollout, 

• ensure that all critical problems identified in pilot testing of the 

electronic disability system components are resolved and that end-toend testing of the interrelated systems is performed;

• ensure that users have approved the software developed and that 

systems have been certified for production; 

• establish a revised time frame for and expedite actions toward finalizing 

AeDib risk mitigation strategies; 
Page 4 GAO-04-466 SSA’s AeDib Strategy

• validate all AeDib cost and benefit estimates; and 

• implement a communications plan that clearly and comprehensively 

conveys SSA’s approach for effectively addressing disability 

stakeholders’ and users’ concerns and ensuring their full involvement in 

the AeDib initiative. 

Agency Comments and 

Our Evaluation

In providing written comments on a draft of this report (reprinted in app. 

II), the Commissioner of Social Security reiterated the department’s 

general disagreement with the findings discussed in our briefing slides. The 

Commissioner added that, in considering our recommendations, the 

agency had not been compelled to change its direction on the AeDib 

initiative.

In particular, SSA continued to disagree with the need for end-to-end 

testing, stating that to perform such testing would delay the agency’s ability 

to realize benefits from this initiative. SSA estimated that following our 

recommendation would have delayed the project, and the resulting 

benefits, by at least 3 years. Given the technological complexity of the 

AeDib project, we remain convinced that end-to-end testing is essential to 

ensuring that the multiple interrelated components of the electronic 

disability system will operate as intended. As our report noted, at the time 

that SSA began national implementation of the electronic disability system, 

it had conducted only limited pilot testing of the Document Management 

Architecture that is fundamental to the electronic disability folder. It also 

had not fully evaluated the results of its pilot tests or ensured that all 

critical problems were resolved. Further, we disagree with SSA’s contention 

that end-to-end testing would delay the AeDib project by 3 years. In 

referring to this time frame, SSA has emphasized the processing of its most 

comprehensive cases—those in which individuals pursue their disability 

claims through all levels of agency appeal—which reportedly take an 

average of 1,153 days to reach a final decision. However, not every case 

that SSA processes requires this level of appeal; thus, some cases can be 

processed in less time. In addition, SSA has reported that, due to backlogs, 

cases that go through all levels of appeal spend nearly 50 percent of that 

time in queue, waiting for agency action. Given these factors, we believe 

SSA has an opportunity to apply a more expedited approach to end-to-end 

testing, focused on evaluating certain claims that require some, but not all 

levels of agency appeal, and processing them outside the normal backlog of 

cases. Without full awareness and resolution of critical problems and endto-end testing, SSA cannot ensure that the electronic disability system 
Page 5 GAO-04-466 SSA’s AeDib Strategy

components will work together effectively. Thus, SSA could incur future 

unanticipated costs to correct the system, and stands to increase its risk of 

not realizing the very benefits that it seeks, by proceeding with AeDib 

without this critical testing.

In addition, SSA said it considered moot our recommendation that the 

agency ensure that users have approved new software and that it certify its 

systems for production, stating that the agency has always performed these 

tasks. In addition, SSA said that our concern in this area had pertained to 

the lack of wet signatures2

 on its project documents. In speaking to this 

issue, our primary concern was not that SSA lacked wet signatures on its 

software development documentation. Rather, during the course of our 

review, agency officials were unable to provide readily verifiable evidence 

that users had approved software developed for the electronic disability 

system, or that systems had been certified for production. At the end of our 

study, SSA provided certain documentation supporting its software 

validation and certification efforts; however, the documents lacked 

sufficient details for us to fully evaluate and reach conclusions about the 

extent to which all software had been validated by the users and certified 

for production. For any project, and especially one of AeDib’s size and 

complexity, having readily verifiable evidence that systems were developed 

in accordance with specified requirements is essential to ensuring that 

users’ needs will be met by the software developed and that systems can be 

certified for production. 

Regarding AeDib program and project risks, SSA stated that it believed the 

success of AeDib over the last 26 months had attested to the agency’s 

diligent and ongoing efforts to mitigate risks. Nonetheless, as our report 

stresses, strategies for addressing program-level and security-level risks 

are needed to help prevent circumstances that could impede successful 

project outcomes. In the absence of explicit mitigation strategies, SSA 

increases the probability of failing to meet project commitments before 

they can manifest themselves in cost, schedule, and performance 

shortfalls. Accordingly, it is essential that SSA establish a time frame for 

and expedite actions toward finalizing risk-mitigation strategies for the 

AeDib project. 

2

A wet signature refers to an original signature written on a piece of paper, as opposed to a 

fax copy or an agreement offered verbally or electronically.
Page 6 GAO-04-466 SSA’s AeDib Strategy

Regarding our recommendation that SSA validate all AeDib cost and 

benefit estimates, the agency reiterated its intent to conduct studies to help 

validate planning assumptions, and noted that work supporting this action 

was under way. 

Finally, SSA stated that it continually reassesses its communications, and 

that while it can always improve, it is satisfied that information is being 

shared, and that stakeholder voices are being heard. Further, SSA stated 

that the stakeholders’ differing opinions about how to design the AeDib 

system do not reflect an issue of communication, but rather, the reality of 

bringing change into a very complex and diverse environment. Our report 

noted that SSA had taken additional actions to work collaboratively with 

stakeholders. Further, we recognize that changing conditions can elicit 

varying opinions regarding the actions being taken on a project. 

Nonetheless, stakeholders continue to voice serious concerns regarding 

the manner in which SSA is communicating with them about AeDib, 

necessitating that the agency remain diligent in pursuing a mutually agreedupon understanding with stakeholders on systems development and 

implementation issues. SSA’s success in fully implementing AeDib depends 

heavily on resolving all outstanding issues and concerns that could affect 

the ultimate use and outcome of the intended electronic capability. By 

implementing a communications plan that clearly and comprehensively 

conveys its approach for effectively addressing disability stakeholders’ and 

users’ concerns and ensuring their full involvement in the AeDib initiative, 

SSA could strengthen the likelihood that it will achieve a successful project 

outcome that effectively meets the needs of all critical players in the 

disability claims process. 

We are sending copes of this report to the Commissioner of Social Security, 

and to the Director, Office of Management and Budget. Copies will also be 

available at no charge on our Web site at www.gao.gov.
Page 7 GAO-04-466 SSA’s AeDib Strategy

Should you have any question on matters contained in this report, please 

contact me at (202) 512-6240, or Valerie Melvin, Assistant Director, at 

(202) 512-6304. We can also be reached by e-mail at koontzl@gao.gov and 

melvinv@gao.gov, respectively. Other key contributors to this report were 

Michael A. Alexander, Tonia B. Brown, Harold J. Brumm, Mary J. Dorsey, 

and Michael P. Fruitman. 

Sincerely yours,

Linda D. Koontz

Director, Information Management Issues 
Page 8 GAO-04-466 SSA’s AeDib Strategy

Appendix I

GAO’s January 30, 2004, Briefing ApApenednxidexiIs

Update on the

Social Security Administration’s

Accelerated Electronic Disability Strategy

Briefing for the Staff of the

Subcommittee on Social Security

House Ways and Means Committee

January 30, 2004
Appendix I

GAO’s January 30, 2004, Briefing

Page 9 GAO-04-466 SSA’s AeDib Strategy

2

Purpose and Outline

Purpose

To provide an updated assessment of SSA’s accelerated electronic disability (AeDib) claims

processing initiative

Outline

• Introduction

• Objectives

• Scope and Methodology

• Results in Brief

• Background

• Objective 1: Strategy

• Objective 2: Software Development and Risk Management

• Objective 3: Cost-Benefit Analysis

• Objective 4: Stakeholder/User Involvement

• Conclusions

• Recommendations

• Agency Comments and Our Evaluation
Appendix I

GAO’s January 30, 2004, Briefing

Page 10 GAO-04-466 SSA’s AeDib Strategy

3

Introduction

Over the years, SSA has faced increasing challenges in ensuring acceptable levels 

of service to the millions of disabled individuals and beneficiaries. A major barrier to 

improving efficiency in the disability determination process has been SSA’s

continuing reliance on paper folders to process and adjudicate claims. 

As a result, SSA is currently pursuing a major systems-related initiative to enhance

its disability claims processing capability. A key goal of this initiative, known as the 

Accelerated Electronic Disability (AeDib) System, is to develop by late January

2004, a disability claims process with the capability to electronically view and share 

claims information among all processing components.

GAO previously reported1 on SSA’s progress in developing the AeDib initiative, 

noting that its strategy involved risks that could jeopardize the agency’s successful

transition to an electronic disability process.

1U.S. General Accounting Office, Electronic Disability Claims Processing: Social Security Administration’s Accelerated Strategy Faces

Significant Risks, GAO-03-984T (Washington, D.C.: July 24, 2003) and U.S. General Accounting Office, Social Security Administration:

Subcommittee Questions Concerning Efforts to Automate the Disability Claims Process, GAO-03-1113R (Washington, D.C.: Sept. 5,

2003).
Appendix I

GAO’s January 30, 2004, Briefing

Page 11 GAO-04-466 SSA’s AeDib Strategy

4

Objectives

As requested by the Chairman of the subcommittee, our objectives were to

• assess SSA’s progress toward and strategy for achieving AeDib,

• determine the adequacy of measures SSA is taking with AeDib to avoid the 

kinds of software development problems that it encountered with its prior 

disability system initiative and to identify and mitigate risks to a successful 

development effort,

• assess the adequacy of SSA’s analysis of AeDib costs and benefits, and

• assess SSA’s consultation with and support from key stakeholders.
Appendix I

GAO’s January 30, 2004, Briefing

Page 12 GAO-04-466 SSA’s AeDib Strategy

5

Scope and Methodology

Analyzed project management and technical documentation describing SSA’s

plans, strategies, and progress related to developing and implementing AeDib

Evaluated the extent to which SSA has implemented measures—including

institutional software capability mechanisms—to avoid the types of software 

development problems identified in prior reviews and established strategies for 

mitigating program and project risks

Assessed the adequacy of SSA’s analyses of AeDib costs and benefits, including

evaluating them against federal guidance and industry best practices and analyzing

the impact of critical pilot evaluation data on costs and benefits

Assessed SSA’s plans and actions to effectively consult with and identify and

resolve issues raised by key disability stakeholders (i.e., state Disability

Determination Services [DDS] offices, medical providers, and SSA field offices)

Conducted site visits at the North Carolina DDS office and SSA field offices in 

Raleigh and Durham to observe pilot tests supporting the electronic folder

capability, and teleconferences with officials in the California and Illinois DDSs and 

the SSA field office in Hillside, Illinois to discuss pilot test progress and results
Appendix I

GAO’s January 30, 2004, Briefing

Page 13 GAO-04-466 SSA’s AeDib Strategy

6

Scope and Methodology

Interviewed officials in selected DDS offices that had been included in our prior 

review2 (Delaware, Iowa, Mississippi, Nebraska, New York, Virginia, and

Wisconsin) to update our assessments of SSA’s involvement of stakeholders in the

AeDib initiative

Interviewed headquarters officials in SSA’s Offices of Disability Programs,

Operations, Systems, Hearings and Appeals, and the Inspector General to obtain 

their perspectives on the AeDib project

Conducted our review in accordance with generally accepted government auditing

standards, from August 2003 through January 2004

2GAO-03-984T.
Appendix I

GAO’s January 30, 2004, Briefing

Page 14 GAO-04-466 SSA’s AeDib Strategy

7

Results in Brief: Objective 1

SSA’s Progress and Strategy for Achieving AeDib

AeDib represents an important step toward more efficient delivery of disability

payments and SSA is in various stages of completing key components of the 

initiative. SSA’s strategy, however, involves risks that could threaten project

success. Specifically, SSA is relying on limited pilot tests to help guide business 

and technical decisions and ensure that technology supporting the electronic

disability folder will work as intended. Further, SSA is beginning its national rollout

without ensuring that all critical problems identified in its pilot testing have been

resolved and without conducting end-to-end testing to evaluate the functionality

and performance of all electronic disability system components collectively. SSA 

maintained that its pilot tests of the electronic disability system components would 

be sufficient to evaluate the system. However, in the absence of end-to-end testing, 

SSA lacks assurance that the interrelated electronic disability system components

will work together successfully.
Appendix I

GAO’s January 30, 2004, Briefing

Page 15 GAO-04-466 SSA’s AeDib Strategy

8

Results in Brief: Objective 2

Previous Problems and Mitigating Risks

SSA has established processes and procedures to guide its software development

activities, including using plans to manage projects and tracking and overseeing

initiatives to measure progress. However, SSA could not provide evidence that it is 

consistently applying these procedures to the AeDib initiative. For example, our 

review found no evidence of approved software validation plans and system

certifications to show that users’ requirements were satisfied. In addition, while

SSA had identified AeDib system and security risks, it had not finalized mitigation

strategies needed to help ensure a successful project outcome. Without

concurrence on software validation and system certifications, or risk mitigation 

strategies, SSA lacks assurance that its system will be acceptable to end users and 

it will not be positioned to effectively prevent circumstances that could impede

project success.
Appendix I

GAO’s January 30, 2004, Briefing

Page 16 GAO-04-466 SSA’s AeDib Strategy

9

Results in Brief: Objective 3

Adequacy of AeDib Cost-Benefit Analysis

SSA has not validated its cost-benefit analysis to ensure the reasonableness of 

estimated AeDib costs and benefits. SSA stated that it would use pilot test results

to validate its cost-benefit estimates; however, its officials have not indicated when

this will be done. Without a validated cost-benefit analysis, SSA lacks assurance 

that its AeDib cost estimates are reliable and that anticipated benefits will be

realized.
Appendix I

GAO’s January 30, 2004, Briefing

Page 17 GAO-04-466 SSA’s AeDib Strategy

10

Results in Brief: Objective 4

Consultation With/Support from Stakeholders

SSA stated that it has increased its communications with AeDib stakeholders/users

and has instituted some mechanisms to facilitate its communications. However,

state DDSs have varying views regarding their involvement. Specifically, while 

state disability offices involved in AeDib pilots agreed that communications have 

improved, a national organization representing these offices continues to have

concern about SSA’s approach. In addition, although the Commissioner has stated 

that SSA is consulting with stakeholders and the medical community, SSA has not 

articulated a comprehensive plan for ensuring that all of their concerns are 

addressed.
Appendix I

GAO’s January 30, 2004, Briefing

Page 18 GAO-04-466 SSA’s AeDib Strategy

11

Results in Brief: Objective 4

Recommendations for Improvement

Given the importance of the AeDib initiative to SSA’s future service delivery

capability, it is essential that the agency satisfy itself that the electronic disability

system will perform as intended with minimal risk before it is deployed nationwide.

Therefore, to help improve the success of this initiative, we are making

recommendations to the Commissioner of Social Security on (1) resolving critical

problems identified and conducting end-to-end testing, (2) ensuring user 

concurrence on software validation and systems certifications, (3) finalizing AeDib

risk mitigation strategies, (4) validating AeDib costs and benefits, and (5) improving

agency communications with stakeholders and users.

SSA officials reviewed a draft of our briefing slides, and their comments have been 

incorporated where appropriate. SSA took issue with many of our findings. For

example, it disagreed with the need for end-to-end testing, and also disagreed with 

our finding that it had not followed all agency processes and procedures for 

software development. We discuss these comments and our response to them in 

the Agency Comments and Our Evaluation section.
Appendix I

GAO’s January 30, 2004, Briefing

Page 19 GAO-04-466 SSA’s AeDib Strategy

12

Background

SSA’s Disability Program

The Disability Insurance and Supplemental Security Income programs are the nation’s

largest providers of federal income assistance to disabled individuals, with SSA making 

payments of approximately $86 billion to about 10 million beneficiaries in 2002.

SSA’s process of approving or denying disability benefits is complex and involves 

multiple partners at both the state and federal levels in determining a claimant’s 

eligibility.

• SSA’s 1,300 field offices and its Office of Hearings and Appeals, along with 54

state DDSs, are the primary players in processing disability claims.

• Physicians and other members of the medical community support disability 

determinations by providing the medical evidence needed to evaluate disability

claims.

To address concerns regarding the disability program’s efficiency, in 1992 SSA initiated 

a redesign of the disability claims process, emphasizing the use of automation to

achieve an electronic (paperless) processing capability. However, SSA encountered

problems with this prior initiative, the Reengineered Disability System (RDS), and in 

1999 suspended the project after approximately 7 years and about $71 million 

reportedly spent on the initiative.3

3U.S. General Accounting Office, Social Security Administration: Technical and Performance Challenges Threaten the Progress of Modernization,

GAO/AIMD-98-136 (Washington, D.C.: June 19, 1998).
Appendix I

GAO’s January 30, 2004, Briefing

Page 20 GAO-04-466 SSA’s AeDib Strategy

13

Background

SSA’s Disability Program

In August 2000 SSA renewed its plan to achieve paperless disability decision

making through the use of an electronic disability folder and automated case

processing systems for disability claims adjudication/review by the end of 2005. In 

the spring of 2002, the Commissioner of Social Security accelerated the strategy to 

more quickly automate the disability claims process. Under the accelerated

strategy—called AeDib—SSA planned to begin implementing its electronic 

disability system by January 2004.
Appendix I

GAO’s January 30, 2004, Briefing

Page 21 GAO-04-466 SSA’s AeDib Strategy

14

Background

Projects Supporting AeDib

SSA is undertaking five key projects to support the AeDib strategy:

1. an Electronic Disability Collect System, giving SSA field offices the capability

to electronically capture and store in an electronic folder information about a 

claimant’s disability;

2. a Document Management Architecture (data repository and scanning and

imaging capabilities) to allow claimant information and medical evidence to

be captured, stored, indexed, and shared electronically among disability

processing components;

3. Internet applications to enable SSA to obtain disability claims and medical

information from claimants via the Internet;

4. a DDS systems migration and electronic folder interface to enhance existing

case processing systems and enable state DDS offices to operate on a 

common platform and share information in the electronic folder; and

5. a Case Processing and Management System for the Office of Hearings and 

Appeals that will interface with the electronic folder and enable staff to track, 

manage, and complete case-related tasks electronically.
Appendix I

GAO’s January 30, 2004, Briefing

Page 22 GAO-04-466 SSA’s AeDib Strategy

15

Background

Projects Supporting AeDib

The AeDib strategy focuses on developing the capability for claimant information

and large volumes of medical images, files, and other documents that are currently 

maintained in paper folders to be stored in electronic folders, and then accessed,

viewed, and shared by the disability processing offices. 

According to SSA, the Electronic Disability Collect System and the Document

Management Architecture are the two fundamental elements needed to achieve the 

electronic disability folder.

SSA is using an incremental, phased approach to implementing AeDib.

The agency planned to rely on pilot tests and evaluations to help guide business

and technical decisions about the electronic disability folder, including critical

decisions regarding the Document Management Architecture.

SSA had planned to complete its pilot tests of the Document Management

Architecture in December 2003, and begin its national implementation of the

electronic disability folder in late January 2004, with scanning and imaging

capabilities and interface software being rolled out over 18 months.
Appendix I

GAO’s January 30, 2004, Briefing

Page 23 GAO-04-466 SSA’s AeDib Strategy

16

Background

SSA’s Approach

AeDib is a major capital investment project, and its February 2003 cost-benefit

analysis estimates 10-year life-cycle costs of about $900 million and benefits of 

approximately $1.3 billion.

SSA reported actual AeDib costs of approximately $91 million for fiscal year 2000 

through fiscal year 2003.

SSA identified two AeDib funding sources for IT hardware, software, and services:

• information technology systems/automation investment fund, and

• limitation on administrative expenses budget.
Appendix I

GAO’s January 30, 2004, Briefing

Page 24 GAO-04-466 SSA’s AeDib Strategy

17

Background

July 2003 Findings

In July 2003 testimony4 we noted that SSA

• had performed important tasks for the initial electronic capability, but had 

substantial work to accomplish in order to begin implementing the electronic

folder by late January 2004;

• was not expected to complete pilot tests that were to provide critical 

information about the electronic folder’s performance until late December

2003 and had not developed a strategy for conducting end-to-end testing to

demonstrate that the individual components would work together reliably;

• had not comprehensively assessed AeDib project risks or developed risk 

mitigation strategies; and

• had taken some steps to involve key stakeholders in the system’s

development, but had not resolved all outstanding concerns that could affect 

the use and ultimate success of the intended electronic capability.

4GAO-03-984T.
Appendix I

GAO’s January 30, 2004, Briefing

Page 25 GAO-04-466 SSA’s AeDib Strategy

18

Background

September 2003 Findings

In September 20035 we reported that SSA 

• may have underestimated the accelerated electronic disability system’s costs

and overstated corresponding benefits in its AeDib cost-benefit analysis, and 

• had applied software development practices that could help the agency avoid

software development problems experienced with its prior electronic disability

initiative—the Reengineered Disability System.

5GAO-03-1113R.
Appendix I

GAO’s January 30, 2004, Briefing

Page 26 GAO-04-466 SSA’s AeDib Strategy

19

Objective 1: Strategy

Progress Update

Since July, SSA has continued its work toward completing the AeDib initiative, and 

is in various stages of development.

• SSA has implemented all planned releases of the Electronic Disability Collect

System (as of January 20, 2004).

• SSA has completed and placed into production three Internet applications to

aid claimants in filing for disability benefits and services online.

• SSA has enhanced the DDS legacy systems by installing hardware upgrades

and moving systems to a common platform; it has also migrated legacy

systems claims processing software in all but 3 of the 54 state DDS offices. 

However, it has yet to complete the legacy system electronic folder interface

software that will enable existing SSA and DDS systems to interface with the

electronic folder. SSA reported that it plans to complete its implementation of 

the legacy system electronic folder interface software by October 2004.

• The Case Processing Management System is being pilot tested in a

standalone environment at five Office of Hearings and Appeals sites.
Appendix I

GAO’s January 30, 2004, Briefing

Page 27 GAO-04-466 SSA’s AeDib Strategy

20

Objective 1: Strategy

Testing

Consistent with generally accepted best practices, SSA should ensure that the 

system it delivers successfully meets key business and technical requirements for 

reliably exchanging data among disability processing components and is protected

from errors and vulnerabilities that can disrupt service.

Accomplishing this requires SSA to conduct complete and thorough testing to 

provide reasonable assurances that systems perform as intended. This includes

tests and evaluations of pilot projects to obtain data on the system’s functional

performance, and end-to-end tests to ensure that the interrelated systems will 

operate together effectively.

SSA planned to rely on the pilot tests and evaluations to guide critical decisions

about its document management technology and verify that this system component

works in the actual user environment. For example, SSA stated that Document 

Management Architecture pilots would be used to test electronic folder interface

and DDS site configurations for the electronic disability system’s national

implementation. SSA and DDS offices stated that they intended to expand the size 

and scope of the pilot tests to analyze the system’s performance.

SSA had planned to complete these pilot tests in December 2003.
Appendix I

GAO’s January 30, 2004, Briefing

Page 28 GAO-04-466 SSA’s AeDib Strategy

21

Objective 1: Strategy

Testing

Since last July, SSA has been pilot testing the Document Management Architecture in 

three locations—North Carolina, Illinois, and California. SSA has been incrementally

providing increased electronic folder functionality.

These pilot tests have demonstrated that the DDSs can view medical evidence captured

in the Document Management Architecture; however, the pilots also identified 

numerous technical/operational problems, which SSA is in the process of resolving:

• DDS pilot systems experienced software glitches (i.e., screen lock-ups, inability to

save highlighted text, and inability to enhance images) after new releases.

• Scanning quality for some documents, such as graphical images, was poor.

• Response times for retrieving medical evidence stored in the data repository was 

slow (California/Midas).

• 2-D barcode resolution diminished with multiple faxing and copying, preventing

effective identification of the case file and the assigned DDS.

• Existing 19-inch monitors proved too small for DDS claims examiners to 

simultaneously view two different documents (e.g., forms and medical evidence)

on split screens.
Appendix I

GAO’s January 30, 2004, Briefing

Page 29 GAO-04-466 SSA’s AeDib Strategy

22

Objective 1: Strategy

Testing

Although the pilot tests are proceeding, they are limited in scope, involving a 

relatively small number of examiners and disability cases:

North Carolina (pilot started July 29, 2003)

• 21 of 254 examiners participating (8 percent)

• Has closed 561 of 1,246 cases received for the pilot

• Closes about 147,000 cases annually

Illinois (pilot started Sept. 8, 2003)

• 15 of 250 examiners participating (6 percent)

• Has closed 282 of 1,165 cases received for the pilot

• Closes about 164,000 disability cases annually

California (pilot started Oct. 15, 2003)

• 2 of 620 examiners participating (0.3 percent)

• Has closed 47 of 160 cases received for the pilot

• Closes about 400,000 disability cases annually
Appendix I

GAO’s January 30, 2004, Briefing

Page 30 GAO-04-466 SSA’s AeDib Strategy

23

Objective 1: Strategy

Testing

Although SSA had planned to expand the pilot tests to additional users, the

technical/operational problems encountered during the pilots (e.g., software glitches 

experienced after new releases) have limited the expansion, and consequently, the opportunity

to analyze system performance under increasingly heavy workloads (e.g., higher transaction

rates.)

The pilot tests, originally scheduled to be completed in December 2003, are also behind

schedule. Further, it is not clear when SSA plans to complete the pilot tests. On January 13,

SSA’s Deputy Commissioner for Systems stated that, beginning on January 20, SSA planned 

to release to the three pilot states, the interface software and enhancements that would provide 

disability users the capability to electronically process an entire disability case.

The Deputy Commissioner added that, barring significant problems, SSA planned to begin the 

national rollout of this functionality in Mississippi on January 26, 2004. Thus, SSA would begin 

its national rollout before fully evaluating pilot test results and ensuring that all critical problems 

are resolved.

Without fully evaluating pilot test results and ensuring the resolution of all critical problems

before national rollout of the electronic disability system, SSA not only risks being unable to 

ensure that the electronic folder will perform as intended, but also risks being unable to rely on 

the new system to sustain current disability workloads.
Appendix I

GAO’s January 30, 2004, Briefing

Page 31 GAO-04-466 SSA’s AeDib Strategy

24

Objective 1: Strategy

Testing

Given AeDib’s technological complexity and multiple interrelated components,

coupled with limited pilot testing of the Document Management Architecture and the 

numerous problems encountered during that testing, end-to-end testing remains

essential to helping SSA reduce system risks prior to national roll-out.

Our prior work has noted the need for such testing to ensure that interrelated

systems that collectively support a core business area or function will work as 

intended in a true operational environment.6 End-to-end testing evaluates both the 

functionality and performance of all systems components, enhancing an

organization’s ability to trust the system’s reliability. 

In addition to its limited pilot testing of the Document Management Architecture,

performance and other problems that SSA has encountered while developing and 

implementing other critical AeDib components further underscore the need for endto-end testing. For example:

6U.S. General Accounting Office, Year 2000 Computing Crisis: A Testing Guide, GAO/AIMD-10.1.21 (Washington, D.C.: November 1998).
Appendix I

GAO’s January 30, 2004, Briefing

Page 32 GAO-04-466 SSA’s AeDib Strategy

25

Objective 1: Strategy

Testing

• since implementing the Electronic Disability Collect System, SSA has encountered

software performance issues, such as slow response times when moving from screen

to screen, which have contributed to longer waits in reception areas, scheduling fewer 

claimant appointments, and backlogs in post-entitlement work; and

• edits for claimant information provided on Internet applications generate additional 

edits after the data are input to the Electronic Disability Collect System. Having to 

respond to the additional edits has contributed to increased SSA field office claims 

representatives’ workloads.

Resolving all critical problems identified in its testing of the AeDib components and then

assessing how these components function in an integrated environment before proceeding to 

national roll-out are essential to gauging the system’s overall successful operation.

Nonetheless, SSA does not plan to conduct end-to-end testing of the electronic disability

system prior to national implementation. SSA has maintained that its pilot tests of each

component of the electronic disability system would be sufficient to evaluate the system’s

functionality and performance.

Without resolution of all critical problems and end-to-end testing, SSA cannot make fully

informed decisions about the system it delivers, risks having a system that will not work 

effectively, and could incur future unanticipated costs for systems adjustments or corrections.
Appendix I

GAO’s January 30, 2004, Briefing

Page 33 GAO-04-466 SSA’s AeDib Strategy

26

Objective 2: Software Development/Risk Management

Previous Problems

Our prior reports7 have noted that effective software development processes and

procedures are essential to producing high-quality software products and ensuring that

the software works as intended.

In response to problems encountered during its prior effort with the Reengineered 

Disability System, SSA established key software process improvement procedures to

guide its software development activities. These procedures include

• using plans to manage projects,

• tracking and overseeing initiatives to measure progress,

• performing quality assurance reviews to ensure compliance with policies and 

procedures, and

• conducting software validation and systems certifications to ensure the

effectiveness of and users’ satisfaction with the software.

Implementation of these processes should help the agency ensure that the system 

being developed is ready for production and will meet users’ requirements.

7U.S. General Accounting Office, Customs Service Modernization: Serious Management and Technical Weaknesses Must Be Corrected,

GAO/AIMD-99-41 (Washington, D.C., Feb. 26, 1999); Customs Service Modernization: Ineffective Software Development Processes

Increase Customs System Development Risks, GAO/AIMD-99-35 (Washington, D.C., February 11, 1999); and Air Traffic Control:

Immature Software Acquisition Processes Increase FAA System Acquisition Risks, GAO-AIMD-97-47 (Washington, D.C., March 21,

1997).
Appendix I

GAO’s January 30, 2004, Briefing

Page 34 GAO-04-466 SSA’s AeDib Strategy

27

Objective 2: Software Development/Risk Management

Previous Problems

SSA is adhering to some of its software development processes and procedures

for the AeDib initiative. For example, it is using plans to monitor software 

development activities and is performing quality assurance reviews. However, the 

agency has not consistently implemented certain key procedures. For example, we 

found no evidence that

• users approved software validation plans to show that they agreed with what

was being produced, and

• all systems currently in production were certified for release, to indicate that 

the systems were developed in accordance with specified requirements.

Without software validation and systems certification, SSA lacks assurance that the 

system is ready for production and will be acceptable to its end users.
Appendix I

GAO’s January 30, 2004, Briefing

Page 35 GAO-04-466 SSA’s AeDib Strategy

28

Objective 2: Software Development/Risk Management

Mitigating Risks

Best practices and federal guidance advocate risk management to identify facts and 

circumstances that decrease the probability of failing to meet project commitments before

they manifest themselves as cost, schedule, and performance shortfalls.

SSA has developed a risk management plan to guide the identification and mitigation of 

AeDib program and project risks. The plan requires SSA to ensure that the necessary risk

assessments and mitigation strategies are developed and implemented to reduce risks and

achieve schedule and performance goals.

SSA has identified system and security risks for the five AeDib projects. For example, it noted

• the need for Document Management Architecture interfaces with other AeDib 

components to be properly defined, tested, integrated, and installed to avoid jeopardizing 

AeDib success, and

• the need for completion and approval of systems security plans for all five projects.

However, SSA has not yet finalized the mitigation strategies needed to help prevent

circumstances that could impede successful project outcomes. SSA had previously stated

that its mitigation strategies would be completed by December 15, 2003. However, according

to agency officials, the draft strategies are currently under review by the Office of the Chief

Information Officer and a date for finalizing them has not been established. Until the 

strategies are finalized, SSA will not be in a position to cost-effectively plan for and prevent

circumstances that could impede project success.
Appendix I

GAO’s January 30, 2004, Briefing

Page 36 GAO-04-466 SSA’s AeDib Strategy

29

Objective 3: Cost-Benefit Analysis

Adequacy

Office of Management and Budget guidance8 states that the purpose of a costbenefit analysis is to promote efficient resource allocation through well-informed

decision making when initiating, renewing, or expanding programs or projects that 

would result in a series of measurable benefits or costs extending for 3 or more 

years.

SSA’s guidance states that a cost-benefit analysis is to be conducted when there is 

a major capital investment, potential major human resources impact, and the 

investment is in competition with other projects for funding and resources.

We previously reported that SSA’s estimates of costs and benefits were based on 

critical assumptions that were not substantiated:

• Critical infrastructure costs, such as outsourced scanning and imaging (keying

and indexing), telecommunications, and disaster recovery, were not included in 

the analysis.

• The estimated electronic receipt of 30 percent of medical evidence by 2004 was 

not based on sound data or tested in pilots.

8Office of Management and Budget Circular A-94 (Revised Transmittal Memo No. 64), October 29, 1992.
Appendix I

GAO’s January 30, 2004, Briefing

Page 37 GAO-04-466 SSA’s AeDib Strategy

30

Objective 3: Cost-Benefit Analysis

Adequacy

Validating the estimates is necessary to review assumptions that were made, and 

to continuously update and ensure that costs and benefits reflect the current

situation and are accurate and realistic.

Our review determined that SSA’s analysis had not been validated to ensure the 

reasonableness of estimated AeDib costs and benefits. SSA agreed with our 

assessment and stated that it planned to use the results of pilot tests to validate 

cost and performance data included in the analysis. However, agency officials

could not state when the validation of costs and benefits would be performed.

Without a validated cost-benefit analysis, SSA lacks assurance that its AeDib cost 

estimates are reliable and that anticipated benefits will be realized.
Appendix I

GAO’s January 30, 2004, Briefing

Page 38 GAO-04-466 SSA’s AeDib Strategy

31

Objective 4: Stakeholder/User Involvement

Consultation and Support

As partners in the disability determination process, stakeholders can offer valuable

insight regarding existing work processes and information technology needs, and 

their involvement in the systems development initiative is essential for ensuring

AeDib’s acceptance and use.

Our work on cross-organizational collaborative projects identified effective 

communication and outreach as a key practice that contributes to success.9 The

research indicated that to achieve this, a specific outreach plan specifying tasks

and mechanisms may be needed.

In July we reported on the need for SSA to resolve stakeholder concerns to ensure 

AeDib’s acceptance and use, and to take additional steps to consult with the

medical community.10 SSA acknowledged the importance of ensuring sound 

relations with stakeholders and the need to take additional actions toward

addressing stakeholders’ concerns.

9U.S. General Accounting Office, Electronic Government: Potential Exists for Enhancing Collaboration on Four Initiatives, GAO-04-6

(Washington, D.C.: Oct. 10, 2003).

10GAO-03-1113R.
Appendix I

GAO’s January 30, 2004, Briefing

Page 39 GAO-04-466 SSA’s AeDib Strategy

32

Objective 4: Stakeholder/User Involvement

Consultation and Support

SSA stated that it has increased its communications about AeDib with state DDSs 

and medical providers, and the agency has established some specific 

communication mechanisms (e.g., ad hoc meetings, Internet television broadcasts,

and an AeDib intranet site). 

State DDSs, however, have varying views and concerns about their involvement

with the AeDib initiative.

• DDSs associated with the Document Management Architecture pilot (North

Carolina, Illinois, and California) agreed that SSA has increased

communications and is working more closely with them.

• Four states that are not participating in the pilot (Iowa, Nebraska, New York,

and Virginia) cited no improvement in SSA’s overall communications with them 

about the development and implementation of the electronic disability system.

• The National Council of Disability Determination Directors, which represents

the DDSs, continues to have concerns about SSA’s communications approach,

stating that the AeDib steering committee is only being used as a vehicle for 

reporting decisions already made, and that improved communication with

DDSs is still needed.
Appendix I

GAO’s January 30, 2004, Briefing

Page 40 GAO-04-466 SSA’s AeDib Strategy

33

Objective 4: Stakeholder/User Involvement

Consultation and Support

While the Commissioner previously stated that SSA was consulting with 

stakeholders and the medical community, agency officials could not articulate a 

comprehensive plan for effectively involving all stakeholders and consulting with 

the medical community.

Without a clear and comprehensive plan for communicating with its stakeholders

and the medical community, SSA risks not obtaining vital end-user buy-in and 

acceptance of the electronic disability system and the medical community’s support 

in providing electronic medical evidence that is crucial to achieving anticipated

benefits.
Appendix I

GAO’s January 30, 2004, Briefing

Page 41 GAO-04-466 SSA’s AeDib Strategy

34

Conclusions

SSA is beginning its implementation of an electronic disability system without fully 

evaluating pilot test results or ensuring that all critical problems have been

addressed, and with no plans for end-to-end testing to guide its system 

implementation. Thus, SSA lacks assurance that the system will operate as intended

in a national environment—making the agency highly susceptible to failure in its 

current effort to achieve an electronic disability claims processing capability.

While SSA has established software process improvement procedures, critical user 

concurrence for all software validations and systems certifications are not evident. 

Without such concurrence, SSA lacks assurance that it is building a system that will 

meet users’ requirements. Further, although SSA had planned to complete strategies 

for mitigating program and project risks by last December, it has yet to finalize them.

SSA is beginning its national rollout of the electronic disability system without a 

validated cost-benefit analysis. Lacking a reliable analysis based on validated data,

the agency is assuming the risk that its anticipated costs may be inconsistent with 

planned spending for the initiative and that anticipated measurable benefits may not 

be realized.
Appendix I

GAO’s January 30, 2004, Briefing

Page 42 GAO-04-466 SSA’s AeDib Strategy

35

Conclusions

Finally, SSA has taken some steps to improve its communications with key AeDib 

stakeholders and users; however, the agency has not yet articulated a 

comprehensive plan for ensuring that all stakeholders and users are fully involved

in the initiative and that their concerns are adequately addressed. State DDSs 

currently have mixed views regarding their involvement, with DDS representatives

continuing to see a need for improved communications with SSA about critical 

AeDib decisions. 
Appendix I

GAO’s January 30, 2004, Briefing

Page 43 GAO-04-466 SSA’s AeDib Strategy

36

Recommendations

Given the technological complexity, scope, and size of the AeDib initiative, it is 

essential that SSA take the necessary steps to avoid the substantial risks of

moving into production with a system that does not perform as intended.

Accordingly, we recommend that the Commissioner of Social Security, before

proceeding with the AeDib national rollout,

• ensure that all critical problems identified in pilot testing of the electronic

disability system components are resolved and that end-to-end testing of the 

interrelated systems is performed;

• ensure that users have approved of software developed and that systems 

have been certified for production;

• establish a revised time frame for and expedite actions toward finalizing AeDib

risk mitigation strategies, 

• validate all AeDib cost and benefit estimates, and 

• implement a communications plan that clearly and comprehensively conveys

SSA’s approach for effectively addressing disability stakeholders’ and users’

concerns and ensuring their full involvement in the AeDib initiative.
Appendix I

GAO’s January 30, 2004, Briefing

Page 44 GAO-04-466 SSA’s AeDib Strategy

37

Agency Comments and Our Evaluation

SSA’s Deputy Commissioner for Finance, Assessment and Management provided

an e-mail message containing the agency’s comments on a draft of the briefing 

slides. In its comments, SSA took issue with many of our findings.

SSA disagreed with the need for end-to-end testing, stating that such testing would

have been too slow for its accelerated strategy. In addition, it stated that examples 

discussed in our slides did not support our position, noting, for example, that the 

Electronic Disability Collect System performance issues highlighted in our slides 

had not appeared until the software was in full production across the country.

We maintain that end-to-end testing is essential for AeDib. The acceleration of 

such a technologically complex initiative, with its multiple interrelated

components, elevates the need for end-to-end testing to help SSA reduce 

system risks, many of which were identified during pilot tests of each 

component. The fact that performance issues did not appear until the

Electronic Disability Collect System software was in production, and that high

edit rates were discovered when data from Internet applications were input

into that system, is further evidence of the need for end-to-end testing to 

provide SSA with the opportunity to evaluate both the functionality and 

performance of all systems components prior to national rollout.
Appendix I

GAO’s January 30, 2004, Briefing

Page 45 GAO-04-466 SSA’s AeDib Strategy

38

Agency Comments and Our Evaluation

Regarding examples of technical problems that we noted for the Document Management

Architecture pilots, SSA stated that (1) it had completed a technical evaluation of the

Document Management Architecture in three pilot states and (2) that four of five examples

identified in our slides were resolved before SSA made a final decision to roll out to

Mississippi. In addition, SSA stated that our finding regarding the 19-inch monitor size

prejudged a conclusion that the screens were too small. 

While SSA stated that it had completed technical evaluations of the Document

Management Architecture pilot efforts in the three pilot offices, it did not provide

evidence of its complete evaluations during our review. Specifically, SSA provided

evaluation results covering tests performed in the North Carolina and Illinois DDSs only

during the July through September 2003 time frame. Further, on January 20, SSA 

provided a software release supporting the electronic folder capability to the pilot offices, 

leaving little time to ensure that the software was fully tested and evaluated and that all 

critical problems were resolved prior to beginning its national rollout. 

In addition, while we acknowledge that SSA may have resolved four of the five problems

noted in our slides, the examples that we cited constituted only a subset of the

numerous problems that were noted in the Document Management Architecture

evaluations provided for our review. Our analysis of the North Carolina and Illinois pilot

evaluations and SSA’s problem tracking system report, and our discussions with 

officials of all three state DDSs participating in the pilots, revealed that, as certain

software performance issues were resolved, other problems appeared, and that DDS

staff participating in the pilots were overwhelmed as workloads and backlogs increased.
Appendix I

GAO’s January 30, 2004, Briefing

Page 46 GAO-04-466 SSA’s AeDib Strategy

39

Agency Comments and Our Evaluation

Finally, it was not our judgment or conclusion that SSA’s monitor size was too 

small. Rather, DDS claims examiners informed us that the 19-inch monitors

were too small for viewing two documents on split screen. For example, during

our visit to the North Carolina pilot site, claims examiners participating in tests

of the Document Management Architecture and Electronic Disability Collect

System stated that the small size of the monitors had affected their ability to 

effectively adjudicate a claim. In discussing this matter with SSA’s Deputy

Associate Commissioner for Systems and Associate Commissioner for 

Disability Programs, both stated that the agency was aware of DDS staffs’

concerns about the size of the monitors and that SSA planned to send an 

ergonomics team to those sites to investigate this and other issues affecting 

staff using the system. At the time of review, the study had not been

completed.
Appendix I

GAO’s January 30, 2004, Briefing

Page 47 GAO-04-466 SSA’s AeDib Strategy

40

Agency Comments and Our Evaluation

In discussing the October 2004 completion date of its legacy systems migration, SSA stated

that, while technically correct, our discussion did not acknowledge that DDS legacy software

migration had been completed in all but 3 of the 54 state DDS offices and that those three

offices would be completed in February 2004.

We have modified our slides to reflect that SSA has completed legacy software

migration in all but three DDS offices. 

SSA also stated that the DDSs have or will shortly have the Electronic Disability Collect

System electronic folder interface (EFI). It noted that its more expansive EFI software is 

currently running in 11 states, and expansion of that will occur in concert with the Document

Management Architecture rollout.

SSA’s recent documentation stated that it has made the Electronic Disability Collect 

System electronic folder interface software available to 15 DDS states, including the 

three Document Management Architecture pilot offices. However, SSA has yet to

complete the necessary DDS legacy systems software—including its critical Document

Management Architecture interface software—to enable these components to process

disability claims. Until this is accomplished, existing disability processing components

will not have the capability to interface with or share information captured in the full

electronic folder.
Appendix I

GAO’s January 30, 2004, Briefing

Page 48 GAO-04-466 SSA’s AeDib Strategy

41

Agency Comments and Our Evaluation

SSA stated that it has followed processes and procedures for software 

development and disagreed with our findings that it did not have approved software

validation plans and had not properly certified all software releases.

While we agree that SSA has established key software process improvement

procedures to guide its software development activities, we found that SSA 

was not consistently adhering to its established procedures. At the time of our

review, SSA could not provide the necessary documentation to demonstrate 

that users had approved of the AeDib software’s being produced. SSA’s

Inspector General had reported the same findings in July 2003, and in 

discussions with us last December, reiterated this continuing concern. 
Appendix I

GAO’s January 30, 2004, Briefing

Page 49 GAO-04-466 SSA’s AeDib Strategy

42

Agency Comments and Our Evaluation

SSA stated that its security risk assessment documentation is no longer in draft

and can be made available to us for review.

SSA’s actions toward finalizing its security level risk assessments for AeDib 

are commendable, and we look forward to reviewing this documentation.

However, our recent reviews of AeDib found that the agency had not finalized

mitigation strategies for its program-level or security-level risk assessments.

SSA officials informed us that the draft mitigation strategies were under review

by the Office of the Chief Information Officer and that no date for finalizing

them had been established. We continue to believe that these strategies are

needed to help prevent circumstances that could impede successful project

outcomes.
Appendix I

GAO’s January 30, 2004, Briefing

Page 50 GAO-04-466 SSA’s AeDib Strategy

43

Agency Comments and Our Evaluation

SSA believes that its 30-percent electronic medical evidence is a viable goal, and 

continued to disagree with our view that its cost-benefit analysis did not consider 

proper infrastructure costs. However, SSA did agree that validation of its cost and 

benefit assumptions is important and stated that it is actively planning that work.

We are encouraged that SSA is planning to validate its cost and benefit

assumptions. In noting SSA’s 30-percent electronic medical evidence goal,

our intent was to highlight the importance of SSA’s validating its cost-benefit 

analysis to ensure the reasonableness of its estimated costs and benefits. The 

assumption that we highlighted was one of many included in SSA’s costbenefit analysis that had not been validated. Further, it continues to be our

position that all relevant costs need to be reflected in SSA’s cost-benefit

analysis. Office of Management and Budget guidance states that cost-benefit 

analyses should include comprehensive estimates of all direct and indirect

costs associated with a project. Without a validated cost-benefit analysis, and 

the inclusion of missing critical cost elements, SSA lacks assurance that its

AeDib cost estimates are reliable and that anticipated benefits will be realized.
Appendix I

GAO’s January 30, 2004, Briefing

Page 51 GAO-04-466 SSA’s AeDib Strategy

44

Agency Comments and Our Evaluation

Finally, regarding our findings on communication and consultation with disability

stakeholders, SSA stated that AeDib information is pervasive and available to all 

stakeholders/users, and that the agency continues to work collaboratively with

them. It added that outreach to external groups will ramp up in relation to national

rollout.

We acknowledge that SSA has made additional efforts to work collaboratively

with its stakeholders regarding AeDib. However, we found during our analyses

of documentation and in discussions with disability stakeholders that there are 

still varying views and concerns about SSA’s approach for implementing

AeDib. Due to the complexity of the AeDib initiative, and the many concerns

that continue to be voiced about SSA’s approach, we continue to believe that

having a clear and comprehensive plan before national rollout is vital to 

ensuring end-user buy-in and acceptance of the electronic disability system, 

and to realizing anticipated benefits.
Page 52 GAO-04-466 SSA’s AeDib Strategy

Appendix II

Comments from the Social Security 

Administration ApendxiI
Appendix II

Comments from the Social Security 

Administration

Page 53 GAO-04-466 SSA’s AeDib Strategy
Appendix II

Comments from the Social Security 

Administration

Page 54 GAO-04-466 SSA’s AeDib Strategy
Appendix II

Comments from the Social Security 

Administration

Page 55 GAO-04-466 SSA’s AeDib Strategy
Appendix II

Comments from the Social Security 

Administration

(310704) Page 56 GAO-04-466 SSA’s AeDib Strategy
GAO’s Mission The General Accounting Office, the audit, evaluation and investigative arm of 

Congress, exists to support Congress in meeting its constitutional responsibilities 

and to help improve the performance and accountability of the federal government 

for the American people. GAO examines the use of public funds; evaluates federal 

programs and policies; and provides analyses, recommendations, and other 

assistance to help Congress make informed oversight, policy, and funding 

decisions. GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

Obtaining Copies of 

GAO Reports and 

Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older 

products. The Web site features a search engine to help you locate documents 

using key words and phrases. You can print these documents in their entirety, 

including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 

daily. The list contains links to the full-text document files. To have GAO e-mail this 

list to you every afternoon, go to www.gao.gov and select “Subscribe to 

e-mail alerts” under the “Order GAO Products” heading.

Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. A check 

or money order should be made out to the Superintendent of Documents. GAO 

also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single 

address are discounted 25 percent. Orders should be sent to:

U.S. General Accounting Office

441 G Street NW, Room LM

Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. General Accounting Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Service Requested

Presorted Standard

Postage & Fees Paid

GAO

Permit No. GI00