SSA Disability Redesign: Actions Needed to Enhance Future Progress
Released: 1999-03-12T00:00:00

Letter

Introduction

The Social Security Administration (SSA) manages two major federal disability programs that provide cash benefits to people with long-term disabilities: the Disability Insurance (DI) and Supplemental Security Income (SSI) programs. The DI program was enacted in 1954 and provides monthly cash benefits to severely disabled workers. SSI was enacted in 1972 as an income assistance program for aged, blind, or disabled individuals whose income and resources fall below a certain threshold. For both programs, disability for adults is defined as an inability to engage in any substantial gainful activity because of a severe physical or mental impairment. Both programs also use the same procedures for determining whether the severity of an applicant’s impairment qualifies him or her for disability benefits. In 1998, almost 11 million people received a total of over $73 billion in disability benefits from these programs.

SSA’s complex process for determining whether an individual qualifies for a disability benefit—the disability claims process—has been plagued by a number of long-standing problems. For example, claimants who have been dissatisfied with the initial determination and have filed an appeal frequently have had to wait more than 1-1/2 years for a final decision. Moreover, as many as two-thirds of these determinations were subsequently allowed by an administrative law judge (ALJ). In the early 1990s, SSA had difficulty keeping up with a rapidly growing workload, and backlogs of appealed cases waiting for a hearing grew. In response to these problems, SSA concluded that minor improvements to the disability claims process would be insufficient and embarked on an effort to fundamentally reengineer, or redesign, its process. In 1994, the agency issued an ambitious plan for redesigning the process within 6 years. However, 2 years into implementing the redesign plan, SSA had not made much progress, and we and SSA concluded that the scope of the plan was too large. The agency reevaluated its approach and, in February 1997, issued a scaled-back plan with revised milestones.

Problems With SSA’s Disability Claims Process Came to the Fore in the Early 1990s

SSA’s disability claims process has long been recognized as complex and fragmented. The decision about whether an individual is disabled is based on standards set forth in the Social Security Act and extensive SSA regulations and rulings. Moreover, disability decisions involve a multilevel process that spans many diverse components, including SSA’s 1,298 field offices, 54 state agencies, and 140 hearing offices. This organizationally complex structure has contributed to a number of problems. For example, through the years a high percentage of claimants who were dissatisfied with their initial determinations received favorable decisions on appeal. Claimants have also waited a long time for final decisions on their eligibility. In the early 1990s, these problems were aggravated by mounting workloads, as applications for disability benefits escalated at the same time that SSA was experiencing a decline in its workforce. This, in turn, caused workloads to back up and increased the time it took claimants to receive decisions on their claims.

The Disability Claims Process Is Complicated and Fragmented

SSA’s disability claims process, which has not changed fundamentally in over 40 years, is inherently complex and fragmented. The process contains several opportunities for appeal, and the organizational unit involved, professional background of the adjudicator, and procedures for making a decision on appeal are all different from those of the initial determination. Each organizational unit has separate lines of authority and goals without responsibility for the overall outcome of the process.

The claims process starts when an individual contacts one of SSA’s 1,298 field offices across the country to apply for benefits. Field office personnel help claimants complete their applications; obtain a detailed medical and work history; and identify other, nonmedical eligibility factors. Field office personnel then forward the claims to one of 54 disability determination service (DDS) agencies that are administered by the 50 states and the District of Columbia, Guam, Puerto Rico, and the Virgin Islands.

Under a unique federal-state arrangement, SSA pays state DDSs to determine whether claimants are disabled. At the DDS, a team consisting of a specially trained disability examiner and an agency physician or psychologist reviews the available medical evidence and gathers additional medical evidence, if necessary. In making the disability determination, the team follows official guidance found in SSA’s Program Operations Manual System (POMS), which is based on applicable laws and SSA’s regulations and rulings and also includes detailed instructions for processing cases. If the claimant is dissatisfied with the initial determination, the claimant may request a reconsideration review within 60 days of receiving the determination. Reconsideration is also performed by the DDSs and is based on the same guidance as the initial determination but is carried out by a new adjudicative team.

If the claimant is dissatisfied with the determination, he or she has 60 days to appeal and request a hearing before an ALJ. ALJs are hearing officers located at 140 hearing offices around the country that are administered by SSA’s Office of Hearings and Appeals (OHA). ALJs review the file to determine if additional medical evidence is needed, conduct a hearing, and render a decision. ALJs conduct de novo hearings; that is, ALJs may consider or develop new evidence and are not bound by DDS determinations. These hearings often present the first opportunity for face-to-face contact between claimants and the individuals deciding their eligibility. In rendering a decision, ALJs do not follow the POMS but rely directly on applicable laws and SSA regulations and rulings. ALJs are subject to the Administrative Procedure Act, which affords them some independence in making a disability decision.

Finally, if the ALJ denies the claim, the claimant has 60 days to request a review by the Appeals Council, an independent review group attached to the OHA and composed of administrative appeals judges. The Appeals Council may decide to dismiss the request for review, grant the request and issue its own decision, or remand the case back to the ALJ. The Appeals Council is the claimant’s fourth and final level of administrative review. Upon exhausting these administrative remedies, the claimant may file a complaint with a federal court. Figure 1.1 shows the four decision points in SSA’s current disability claims process.

SSA’s approach to reviewing the quality of the disability decision reflects the complex and fragmented nature of the process. As we have previously reported, current quality assurance reviews focus on DDS determinations and ALJ decisions in isolation from one another, and the approach for reviewing DDS determinations differs from the approach for reviewing ALJ decisions. Reviews of DDS determinations are conducted by staff from SSA’s Office of Quality Assurance (OQA). These reviews focus heavily on DI claims that have been allowed. In conducting their quality review, OQA staff use the same approach, policy, and procedures that the DDSs use in reaching a determination; that is, they rely on the POMS. In contrast, only a small number of ALJ allowance decisions are selected for review by SSA’s Appeals Council. For the most part, reviews of ALJ decisions predominantly consist of reviews of claims denied by ALJs and appealed by claimants to the Appeals Council. In reviewing ALJ decisions, the Appeals Council relies on the same laws and SSA regulations and rulings as those used by ALJs.

Long-Standing Problems Are Associated With the Claims Process

SSA’s disability claims process has long suffered from problems associated with its complexity and fragmentation. Among these problems are the high allowance rates by ALJs of appealed DDS determinations. In fiscal year 1993, before SSA issued its redesign plan, 68 percent of determinations that were appealed received favorable decisions at the hearing level. High ALJ allowance rates have been attributed to a number of factors. According to SSA, an ALJ might arrive at a different decision than a DDS because the claimant’s condition has worsened, or because ALJs are more likely than DDS decisionmakers to meet with claimants face-to-face, and thus have access to more or different information. However, SSA studies have also found that DDS and ALJ adjudicators often arrive at different conclusions even when presented with the same evidence. Disability decisions require difficult judgments, and adjudicators sometimes reach different conclusions. Further, DDS and ALJ adjudicators use medical expertise differently and rely on different documents for guidance when making decisions. Finally, training has not been delivered consistently or simultaneously to all groups of decisionmakers.

This high rate of allowances at the hearing level has raised questions about the fairness, integrity, and cost of SSA’s disability program. In fiscal year 1998, the cost of making a determination at the DDS level was $547 per case, while the cost of an ALJ decision was an additional $1,385. In general, the costs of administering these disability programs reflect the demanding nature of the process: in fiscal year 1998, SSA spent about $4.3 billion, or almost 66 percent of its administrative budget, on the disability programs, even though disability beneficiaries are only 21 percent of the agency’s total number of beneficiaries.

Another long-recognized problem with SSA’s claims process is that many claimants must wait a long time for their final decisions. Because of the multiple levels and decision points in the process, a great deal of time passes while a claimant’s file is passed from one employee or office to another. Delays are also caused by the need to obtain extensive medical evidence from health care providers to document the basis for disability.One SSA study conducted in 1993 showed that an average claimant waited up to 155 days from initial contact with SSA until receiving an initial determination notice, during which time 16 to 26 employees might have handled the claim. Only 13 hours of these 155 days were spent on “task time”—that is, time spent working directly on the case. Further, the study found that it could take up to 550 days from initial contact to receipt of a hearing decision, with only 32 hours of this time spent on task time. As a result of these multiple handoffs and the general complexity of the process, SSA believes claimants do not understand the process and have had difficulty obtaining meaningful information about the status of their claims.

The Mounting Disability Workload in the Early 1990s Exacerbated Problems

In the early 1990s, SSA’s problems with its disability claims process came to the fore as the growing workload placed additional pressure on SSA’s already inefficient process. The number of initial claims had been rising steadily, but it increased dramatically between fiscal years 1991 and 1993—from about 3 million to 3.9 million, or almost 32 percent.Moreover, future increases were expected. At the same time, SSA had to manage this growing workload with staffing levels that had been falling since the 1980s. As a result, SSA’s disability workload began to accumulate during this period. Most dramatically, the number of pending hearings almost doubled between 1991 and 1993—from 183,471 to 357,564.

SSA’s Initial Efforts to Redesign the Process Moved Slowly

To address these long-standing problems and dramatically improve customer service, SSA embarked on a plan in 1994 to radically redesign its disability claims process by completing 83 initiatives over 6 years. We concluded in a 1996 report, however, that 2 years into the plan, SSA had yet to achieve significant progress. SSA’s slow progress was due in part to the overly ambitious nature of the redesign plan, the complexity of the redesign initiatives, and inconsistent stakeholder support and cooperation.

SSA’s Initial Redesign Efforts

Concerned about the inefficiency of the disability claims process and its effect on the quality of service to the public, SSA’s leadership decided in 1993 that the agency needed a strategy for radically improving the process. SSA reviewed reengineering efforts and approaches in other organizations and concluded that process reengineering was critical to achieving its strategic objective of providing world-class service. SSA then created a Disability Process Redesign Team composed of 18 SSA and state DDS employees with varied experience and backgrounds and charged it with fundamentally rethinking and redesigning SSA’s claims process from start to finish. Consistent with commonly held reengineering principles, the team collected extensive information on the process itself and options for improving it. These efforts culminated in a redesign proposal that was widely distributed throughout SSA and the state DDSs and to interested public and private individuals and organizations to solicit comments, concerns, and ideas for improvement. The proposal was also published in the Federal Register, and a comment period elicited 6,000 written responses, which were considered as SSA finalized its initial redesign proposal.

In September 1994, SSA issued its vision for fundamentally redesigning the disability claims process. SSA’s vision included five objectives for the redesigned process: (1) making the process “user-friendly,” (2) allowing claims that should be allowed at the earliest possible level, (3) making the disability decision as quickly as possible, (4) making the process efficient, and (5) providing a satisfying work environment for employees. SSA’s vision was based on more consistent guidance and training for all adjudicators; an automated and simpler claim intake and appeal process; a single, comprehensive quality review process; and a simplified method for making disability decisions. From the claimant’s perspective, the redesigned process was to offer a range of options for filing a claim; provide a single point of contact; and have fewer decision points, as shown in figure 1.2.

SSA had high expectations for its proposed redesigned process. The agency projected that the combined changes to the process would, by fiscal year 1997, result in a 25-percent improvement in productivity and customer service over projected fiscal year 1994 levels, and a further 25 percent by the end of fiscal year 2000—all without a decrease in decisional accuracy. SSA did not expect the overall redesigned process to alter total benefits paid to claimants, but it estimated that the changes would result in administrative cost savings of $704 million through fiscal year 2001, and an additional $305 million annually thereafter.

After putting forth its broad vision, SSA issued in November 1994 a more detailed plan for developing, testing, and finally implementing proposed disability process improvements. The plan originally included 83 initiatives to be accomplished over 6 years. SSA recognized in its implementation plan that most, if not all, of the proposed process changes were interdependent, and that the development, testing, and implementation of related changes would need to be properly sequenced. For example, SSA recognized that all activities and associated benefits were dependent on improvements to its computer system, which were not expected to be completed until the end of the 6-year time frame.

Progress on the Initial Redesign Plan Was Slow

In 1996 and 1997, we issued several reports that raised concerns regarding SSA’s redesign effort. These concerns included, among other things, a lack of progress and demonstrable results. For example, we reported that SSA had not fully completed any of the 38 near-term initiatives it had hoped to accomplish in the first 2 years. As a result, SSA did not have any concrete results available to demonstrate the efficacy of its proposed initiatives.

SSA’s slow progress was due in part to its overly ambitious redesign plan and the complexity of some of its redesign initiatives. We reported that SSA did not follow best practices when it decided to take on a large number of initiatives concurrently. Specifically, we reported that successful reengineering calls for focusing on a small number of initiatives at one time, whereas SSA decided to tackle 38 initiatives in the first 2 years of its redesign effort. Moreover, some of these initiatives were large in scope and very complex. For example, scheduled implementation of SSA’s large and complicated initiative for redesigning its computer system was delayed because of problems identified during testing.

Some aspects of SSA’s redesign plan faced considerable opposition. As part of its redesign effort, SSA had identified over 100 individual groups—both internal and external to SSA—as having a stake in the process and whose involvement was, in many cases, critical to the entire disability claims process. These stakeholder groups—which included various SSA employee unions and associations, state entities and organizations, congressional committees, other federal agencies, and advocacy groups—had a wide variety of views on SSA’s plan, and some opposed specific initiatives. For example, SSA’s plan called for a new position—a disability claims manager (DCM)—that would combine the duties of field office and DDS personnel into one position. The DCM represented significant change to the current process, and SSA faced numerous challenges in obtaining stakeholder cooperation for this key initiative.

In light of these difficulties and in order to increase SSA’s chance of success, we recommended in our December 1996 report that SSA reduce the scope of its redesign effort by focusing on those initiatives considered most crucial to improving the process and testing those initiatives together, in an integrated fashion, at a few sites. In another 1996 report, we recommended that, concurrent with the first phase of its DCM test, SSA test alternatives that we believed were more feasible and compare their relative costs and benefits with those of the DCM before deciding to increase the number of DCM test positions. Later, we supported SSA’s redesign efforts associated with its initiative to improve the consistency of disability decision-making and recommended, among other things, that SSA establish a performance goal for this key redesign initiative.

SSA Scaled Back Its Redesign Plan

As a result of our input, the overall lack of progress, and stakeholder concerns, SSA reassessed its approach to redesign and issued a revised plan in February 1997. The new plan focused on eight key initiatives, each one intended to effect a major change to the system. The plan also included updated tasks and milestones for each key initiative and expanded the time frame for the entire redesign project from 6 to 9 years, ending in 2003. The eight initiatives and their milestones are described in figure 1.3.

As shown in figure 1.3, five of the eight initiatives had relatively near-term deadlines—that is, before the end of fiscal year 1998—for completing a key test or beginning implementation. Two of these initiatives involve testing new positions and, if test results warrant, implementing new positions on a stand-alone basis—that is, independently of other, related initiatives. One new position, the single decision maker (SDM), would expand the DDS disability examiner’s authority to determine certain claims without relying on the DDS physician; the SDM would instead use the physician as a consultant on an as-needed basis. The SDM was expected to make the initial determination process faster and more efficient by eliminating handoffs to DDS physicians in those cases in which the appropriate determination was clear. Another new position, the adjudication officer (AO), would review cases that were appealed to the hearing level. The AO was to help claimants understand the appeals process and would have authority to grant disability benefits in cases in which it was clear that the claim merited a fully favorable decision. In all other cases, the AO was to make sure that all pertinent information was included in the case file and was fully explained, thus facilitating its use by the ALJ at the next level of appeal. By performing these tasks, the AO was expected to improve customer service and make the appeals process faster and more efficient.

A third near-term initiative is the full process model (FPM) test. The FPM combines five proposed changes into a single test to investigate their interactive effects on creating a more efficient process and better customer service. The five tested changes are (1) creating the SDM position; (2) creating the AO position; (3) establishing a new predecision interview, in which the SDM would interview claimants when the evidence did not support a fully favorable determination in order to obtain any additional information before making the final determination; (4) eliminating the reconsideration step; and (5) eliminating the Appeals Council step—that is, removing the claimant’s option to request a review by the Appeals Council of an ALJ decision.

The two other near-term initiatives—process unification and quality assurance—are considered essential elements for achieving correct decisions in the new disability claims process. The intent of the process unification initiative was to achieve similar results on similar cases at all stages of the process. To this end, SSA planned a number of activities, including conducting ongoing training; clarifying policies; and developing unified guidance, called a single presentation of policy, for making disability decisions across all levels of the process. SSA also planned to complete eight additional subinitiatives—all designed to help reduce inconsistencies in decision-making between the DDS and ALJ levels.

SSA’s quality assurance initiative included near-term activities in two areas. First, as part of each of the other major redesign initiatives, SSA planned to develop and test “in-line” quality assurance approaches—such as training, mentoring, and peer review—in order to build quality into the process before decisions are made. Second, SSA planned to develop and test a single “end-of-line” quality review mechanism that covered the entire adjudicatory process from beginning to end and provided data on problems or failures in a component’s in-line quality assurance process. Appendix I provides additional information on SSA’s five near-term redesign initiatives.

Objectives, Scope, and Methodology

The Chairman of the House Subcommittee on Social Security, Committee on Ways and Means, asked us to (1) assess SSA’s efforts to redesign its disability claims process and (2) identify any actions needed to better ensure future progress. We agreed to focus our work on the five initiatives in SSA’s scaled-back plan that have relatively near-term dates for testing, implementation, or both: the SDM, AO, FPM, process unification, and quality assurance initiatives.

In assessing SSA’s redesign experience, we obtained documents from and interviewed SSA officials responsible for planning, managing, and evaluating redesign efforts. We visited several DDS and hearing office test sites and interviewed test participants and managers in Richmond, California; Brooklyn, New York; Raleigh, North Carolina; and Providence and Warwick, Rhode Island. We also interviewed SSA regional officials with responsibility for overseeing or coordinating redesign efforts within their regions as well as representatives of nine major stakeholder groups to obtain their views on SSA’s specific initiatives and general approach for redesign. Finally, we reviewed the literature and interviewed experts on business process reengineering.

We conducted our work between August 1997 and November 1998 in accordance with generally accepted government auditing standards with the following exception: we did not independently verify agency data, including test data on redesign initiatives. We did obtain information from SSA on steps it took to obtain and verify the test data and any problems associated with them. We have noted our concerns regarding the validity and reliability of the data in the report, where appropriate. We obtained comments from SSA officials responsible for the redesign tests, which we have summarized in chapter 4.

Progress Under Revised Plan Hindered by Overly Ambitious Strategy and Problems With Test Design

After 4 years of redesign efforts, SSA has made only limited progress toward improving its disability claims process. While narrowing the focus of its redesign plan has helped, SSA has continued to miss milestones and has not clearly demonstrated that the proposed initiatives would significantly improve the current process. As a result, SSA has had to defer service improvements and reduce estimated savings. The agency’s limited progress has resulted, in part, from SSA’s overly ambitious strategy for testing and implementing its redesign initiatives. Conducting a number of tests and other redesign activities simultaneously proved to be too difficult to keep on track. In addition, problems with SSA’s approach to designing and managing its tests of new initiatives contributed to marginal and inconclusive test results and made it more difficult for SSA to discern how a tested initiative would operate if implemented on a widespread basis.

Progress Under Revised Plan Has Been Limited

SSA has made only limited progress in improving its disability claims process, despite having fewer initiatives in its revised redesign plan than in the original plan. The agency has not met most of its adjusted milestones for testing and implementing its five near-term initiatives. Moreover, results from SSA’s stand-alone tests of two new decisionmaker positions, the SDM and the AO, were not compelling and did not support broader implementation. Therefore, SSA decided to wait for preliminary results of its integrated test, which has in fact produced some promising results. In addition, SSA has made progress under its process unification initiative, such as providing training and clarifying policy, and agency officials believe the actions taken thus far have had a positive effect on customer service. Overall, however, as a result of missed milestones and disappointing test results, SSA has deferred many other process improvements and reduced its redesign expectations for administrative savings.

Delays Continue Under Revised Plan

Even under its scaled-back plan, SSA continues to experience delays. As of October 1998, the agency was behind schedule on all five of the plan’s near-term initiatives. After more than 3 years of testing, SSA had yet to complete its test of the AO decisionmaker position and, for reasons discussed in the next section, SSA has delayed its decision on whether to implement both the SDM and AO decisionmaker positions. Also, SSA did not complete its assessment of the FPM test results in fiscal year 1998 as scheduled.

SSA has completed some of its planned activities under its process unification initiative, but it has missed other key implementation deadlines. The agency has clarified key policies and, since 1996, has issued policy instructions in the same format for all adjudicators. SSA has also provided an unprecedented training program involving 15,000 decisionmakers and quality reviewers from key components of the disability claims process and has adopted process unification principles for its ongoing training program by providing the same training to all adjudicators. However, SSA has experienced delays in several other planned activities. For example, the agency is behind schedule on a test to study the effect of requiring DDS adjudicators to more fully document the rationale they used in making particular disability determinations. SSA hopes this more detailed explanation will reduce decisional inconsistencies.

SSA has begun work on its quality assurance initiative, but this effort has also been delayed. As part of its tests of other redesign initiatives, SSA has been exploring “in-line” quality assurance approaches—such as training and mentoring—that are intended to build quality into the process before decisions are made. SSA planned to institute these practices nationwide when it implemented the other redesign initiatives; however, delays in implementing the initiatives have delayed the widespread use of these quality assurance practices. In addition, SSA is more than a year behind in developing a single end-of-line review mechanism. The agency planned to develop one quality standard for its end-of-line reviews in fiscal year 1997 and to test its use in fiscal year 1998. However, as of the end of fiscal year 1998, SSA had not reached internal agreement on what that single quality review standard should be.

Key milestones and the status of SSA’s five near-term initiatives are summarized in table 2.1. Additional information on SSA’s efforts to meet near-term milestones—including specific actions taken to date and the nature and extent of delays—can be found in appendix II.

Efficacy of Redesign Initiatives Has Not Yet Been Fully Demonstrated

As of October 1998, SSA had not clearly demonstrated that its proposed changes would achieve the desired improvements in the disability claims process. SSA had expected the new SDM and AO positions to significantly improve the efficiency and processing time without sacrificing the quality of decision-making. However, results from the stand-alone tests of these positions have been largely disappointing and, in some cases, inconclusive. As a result, SSA decided to postpone implementation decisions on these two initiatives until results from the agency’s integrated FPM test were available.

As an example, SSA had hoped that permitting the SDM to make disability determinations independently, using the DDS physician only on an as-needed basis, would reduce the time spent on the determination process. However, early test results revealed that the SDM position would, on average, reduce by only 1 day the time claimants waited for an initial determination and by only 3.6 minutes the time personnel actually spent working on the case. Moreover, SDM determinations for certain impairment categories were less accurate than under the current system. However, early results from the test of the FPM initiative, where the SDM was tested with other process changes, have shown more promise for the SDM. SSA’s final evaluation of the FPM test for four of the five process changes will not be available before October 1999.

Table 2.2 shows the final or most recent results of the tests of the three initiatives. Appendix III contains more detailed information on test results for these three initiatives.

SSA has not made enough progress on its two other near-term initiatives, process unification and quality assurance, to fully assess their efficacy. Although SSA has not completed many of its planned measures for these two initiatives, some of the early process unification actions may have had a positive effect on customer service. SSA reported that it has accurately paid benefits to approximately 90,000 people 500 days earlier in the process than otherwise might have been the case. While SSA generally did not test these process unification initiatives before implementing them, officials believe that the increase in allowances made earlier in the process is in large part due to the agency’s process unification efforts. At the same time, these officials noted that other factors can influence allowance rates. Therefore, without conducting carefully structured tests, it is difficult to isolate the effects of actions taken by the agency.

Delays and Disappointing Test Results Led to Decreased Expectations for Redesign

As a result of the delays and disappointing test results, SSA has decreased projected administrative savings and postponed the anticipated date for realizing any savings. In 1997, SSA projected savings of 12,086 staff-years for fiscal years 1998 through 2002, resulting from implementing several process changes. SSA planned to use some of these staff-year savings to help with other workloads. Instead, in 1998, SSA both decreased its savings projections and postponed the date it expected to realize savings, which changed its projected staff-year savings to 7,207 through fiscal year 2003. Table 2.3 shows how SSA’s projected staff-year savings changed from its 1998 to its 1999 President’s budget.

More importantly, test results have not provided a compelling case for SSA to make these changes and thereby improve customer service as quickly as it had hoped. Overall processing times have not significantly changed since the beginning of redesign; that is, while processing times have decreased at the initial level, they have increased at the ALJ level. On the other hand, more allowances are being made earlier in the process, and SSA attributes this to its process unification efforts, which were planned to improve customer service without significantly increasing the overall cost of providing benefits.

Limited Progress Is Related to SSA’s Testing and Implementation Strategy

SSA’s difficulties in achieving appreciable improvements in its disability claims process have been caused, in part, by the scope of SSA’s revised plan and the agency’s strategy for testing its proposed process changes. Much like its original plan, SSA’s February 1997 plan was designed to achieve quick and major improvements on many fronts simultaneously in response to the pressing problems with the claims process. However, as with the original plan, SSA’s revised plan proved to be too ambitious and difficult to manage within established time frames. Moreover, SSA’s decision to conduct stand-alone tests contributed to marginal SDM and AO test results, and weaknesses in how SSA designed and managed its AO test contributed to unreliable AO test results. Finally, difficulties SSA experienced with testing changes in an operational environment raised questions about how a tested initiative would operate if implemented.

SSA’s Revised Plan Still Proved Too Ambitious

Like its original plan, SSA’s revised plan was designed to provide a number of near-term, visible improvements, while also laying the foundation for long-term changes. To accomplish this, SSA acted to make progress on a number of fronts simultaneously. For example, hoping to alleviate growing workloads at the appellate level, SSA began testing and planned to implement the AO position independently of other initiatives, even though certain changes that could support the position, such as the redesigned computer system, were still being developed. SSA also began testing and planned to implement the SDM position by itself because SSA believed it could achieve quick and decisive improvements through this position. The agency believed these quick improvements would build momentum for redesign and increase stakeholder support. While the AO and SDM tests were still ongoing, SSA began its FPM test, which investigated the interactive effects of five process changes together: the new SDM and AO decisionmaker positions, the predecision interview with the claimant, eliminating the reconsideration step, and eliminating the claimant’s option to request a review by the Appeals Council. In addition, SSA began testing and developing subinitiatives under its process unification, quality assurance, and the three remaining longer-term initiatives. Given the urgent need to fix the process, SSA considered this ambitious approach appropriate as well as consistent with reengineering theory. At that time, reengineering theory generally called for short time lines for testing and implementing major process changes.

In addition to this multifaceted approach, SSA decided for several reasons to conduct its tests of the proposed redesign changes at many sites and to involve numerous participants. First, officials believed this approach would build trust among employees and other stakeholders, who feared that redesign would negatively affect them. Second, SSA believed it needed to use a large number of test cases to produce statistically valid information in key areas. For example, SSA wanted sufficient data to determine the impact of redesign initiatives on the accuracy of SDM determinations in each major category of impairment. Finally, SSA wanted to have enough data to demonstrate the impact that changes to the process would have on benefit outlays. SSA officials told us that Office of Management and Budget officials were concerned that the proposed changes to the claims process could result in large, unanticipated increases in benefit outlays. Because of the size of the disability programs, even a small increase in the percentage of claimants awarded benefits can result in a significant increase in program costs. For example, SSA officials have roughly estimated that a 1-percent increase in allowances in the disability programs for a period of 10 years could result in an increase of $11 billion in the total benefits paid to beneficiaries—that is, program costs—during that same period.

As a result of SSA’s decision to conduct many tests simultaneously, at one point SSA was testing four near-term initiatives and training test participants for another, longer-term initiative, the DCM. These tests were being conducted at more than 100 sites and involved over 1,000 participants. Table 2.4 shows SSA’s testing schedule, including numbers of sites and participants.

Despite SSA’s good intentions, its scaled-back plan still proved to be too ambitious, and the agency had difficulty keeping it on track. Conducting several large tests that overlapped in time consumed a great deal of management attention and resources. In addition to developing the test plan, implementing and monitoring the test, and collecting and analyzing data, each test involved negotiating and coordinating activities with test sites, test participants, employee unions, and other stakeholders. This large array of testing and evaluation activities made it difficult for SSA to stay on schedule and simultaneously maintain sufficient focus on other redesign efforts—such as process unification and quality assurance.

Unrealistic milestones for specific initiatives also contributed to missed deadlines. For example, SSA allowed itself only 17 months to conduct the FPM test and assess the results, even though it can take up to 21 months for a test case to make its way through the entire disability claims process. In addition, SSA’s milestones for the eight process unification subinitiatives were probably too ambitious (they did not include sufficient time to conduct needed tests or make procedural changes), especially given the overall magnitude of SSA’s redesign efforts and the complexity of the problems these subinitiatives are intended to address. Moreover, some of the factors that contributed to differences between decisions made by the DDS adjudicators and the ALJs have evolved over a number of years and involve sensitive legal issues.

Finally, other competing workloads placed considerable strain on SSA’s ability to manage the overall redesign effort. Besides the redesign initiatives, disability program officials and staff had to cope with additional unanticipated duties and responsibilities. For example, legislation that reformed the nation’s welfare program in 1996 also required that children receiving benefits under the SSI program meet a stricter definition of disability than had been applied in the past. As a result, during fiscal year 1997, when many redesign initiatives were being tested, SSA’s disability staff also had to plan and execute a review of the eligibility of over 288,000 children receiving SSI benefits.

Decision to Conduct Stand-Alone Tests Contributed to Marginal Results

SSA’s decision to test its AO and SDM initiatives independently from related initiatives contributed to the disappointing test results. SSA conducted these stand-alone tests because it wanted to institute these two positions quickly. However, as initially envisioned, these initiatives were expected to result in process improvements and administrative savings in concert with other initiatives. Tested alone, these positions did not demonstrate potential for significantly improving the process.

To illustrate, at the very early stages of its redesign effort, SSA developed expectations for AO productivity assuming that the AO would be operating in a completely redesigned environment. However, the AO test did not include supporting initiatives, such as a redesigned computer system, and, consequently, AO productivity was far below SSA’s expectations. Similarly, the SDM was expected to be operating in a redesigned environment that included, among other changes, the new responsibility of conducting a predecision interview with claimants. The results of the stand-alone SDM test indicated a decline in the accuracy of initial determinations; on the other hand, the integrated FPM test indicated that adding the predecision interview to the SDM’s responsibilities may improve accuracy, as compared with the current process. This improved accuracy may have resulted because SDMs collected more or better data during the predecision interview or because SDMs performed their job more thoroughly in preparation for a meeting with the claimant. While SSA could not have predicted the precise impact of not including a particular process change in its stand-alone tests, the agency understood from the outset of its redesign effort that proposed changes were closely linked and that they depended on each other—especially on computer supports—to dramatically improve the process.

Overall, the decision to conduct stand-alone tests caused delays, did not result in the efficient use of resources, and did not achieve the agency’s goal of quickly building trust and enthusiasm among those who resisted the changes. For example, despite the improved performance of the SDM in the FPM test, pockets of opposition to the SDM, particularly among groups representing some DDS physicians, still existed. While groups that perceive themselves to be negatively affected by change may not be swayed on the basis of clear and positive test results, marginal or inconclusive results provide detractors with a firmer basis to oppose change.

Test Design and Management Problems Contributed to Unreliable AO Test Results

The AO test suffered from a number of design and management flaws that raised questions about the reliability of certain test results. For example, to ensure that AO sites were staffed with the best employees possible, SSA selected test participants from a national pool and temporarily relocated them to their preferred locations. Since the test lasted some time, many of these employees decided to return to their home units, and SSA had to replace them with new, less experienced employees. Replacing test participants created instability in the test environment that negatively affected the test results. In addition, SSA did not arrange for AOs to have necessary supports (such as computers, clerical assistance, supervisors, or feedback from ALJs), which contributed to poor results. Consequently, SSA took steps to refine the AO test and provided additional supports, including training and feedback, to test participants. Accuracy and productivity subsequently improved, although productivity has not improved to the level originally expected by SSA.

Despite these improvements, other problems with how the test cases were handled made it difficult for SSA to assess the efficacy of the AO position. Under the proposed process, an AO cannot deny a claim, so, when an AO does not allow a case, the AO is then required to make sure that all pertinent information is properly arranged in the case file and to prepare a thorough explanation of all medical evidence so that the case can move expeditiously to an ALJ hearing. To fairly assess the impact that the AO had on processing time at the appellate level, SSA planned to compare cases prepared by AOs with a small group of control cases in which no AO had been involved. The two groups were to be handled in a comparable manner; for example, both sets of cases were to be promptly scheduled for hearings. However, in many instances OHA staff did not follow instructions concerning how the cases were to be handled. Since the number of control cases was relatively small, when the improperly handled cases were excluded from the analysis, the number of useable control cases was too small to permit a valid comparison. In addition, SSA did not design its test to determine the overall impact of the AO-prepared cases on the quality of decisions at the next appellate level. Without reliable data on its control group or sufficient data on the impact on quality, SSA could not fully assess the effect of the AO position on the claims process.

Valid Tests Were Difficult to Conduct in SSA’s Operational Environment

SSA’s other tests—of the SDM and FPM—suffered from design problems that stemmed largely from difficulties with trying to conduct a test in a “real world” operational environment. While the SDM and FPM tests provided information and insight into the efficacy of these two concepts, operational limitations made it difficult for SSA to conduct a statistically valid test and conclusively demonstrate how a tested initiative would operate if implemented.

To a lesser extent than with the AO test, SSA’s test of its SDM initiative also provided incomplete information and limited assurance that the initiative would perform as tested. For example, under the current process, 50 percent of DDS allowance determinations are reviewed by regional quality assurance staff, and errors are returned to the adjudicator for correction. However, under the SDM stand-alone test, 100 percent of all determinations—allowances and denials—were reviewed by SSA quality assurance staff and returned for correction. As a result, a large number of cases were returned to SDM adjudicators even though, on average, there was not a large difference in error rates between the SDM and the current process. SDM test participants and other DDS officials told us that this 100-percent review probably caused test participants to rely more heavily on agency physicians than they might have otherwise. In addition, because SSA does not have administrative control over state DDS programs—which are under the direction of state governors—the agency was not able to select a strictly random group of test sites or participants; nevertheless, SSA officials believe that the participant selection methods they used came as close to random as possible, given the present constraints. Moreover, because workloads and production capacity varied at the sites, SSA could not dictate the number of test cases at each site and was therefore unable to distribute the test caseload in a representative manner. Finally, the test was not initially designed to collect data on test cases as they moved beyond the initial determination level to the appeals level—data that would have helped determine the impact of the SDM on overall appeal rates, processing time, efficiency, and quality of determinations. SSA has since modified its approach to collect some of this information.

In designing the FPM test, SSA overcame some, but not all, of the problems experienced with previous tests. For example, SSA was able to persuade states that it believed were nationally representative, on the basis of an analysis of state characteristics, to participate in the test. SSA also decided to track cases through the entire disability claims process, rather than through the initial determination level only. To further ensure a sound test design, SSA hired a consulting firm to independently evaluate the design of the test. While the firm found the test design to be basically sound, it made several suggestions to improve the test and better ensure stakeholder confidence in the validity of the test results. SSA was not able to make all the recommended changes, however. For example, because of state union-management agreements, SSA was unable to obtain data on the qualifications of employees to ensure that test participants were representative of all employees, as recommended by the consultant. In addition, contrary to the consultant’s recommendation, SSA did not mitigate the impact of the 100-percent review of SDM determinations for quality, which may lead to some of the same problems experienced with the SDM stand-alone test.

Actions Needed to Improve Chances for Future Progress

While SSA has experienced, and continues to face, many difficulties with its redesign effort, the agency can still take actions to increase its chances for future progress. As SSA continues its redesign work, it has an opportunity to apply lessons learned from its 4 years of reengineering experience, as well as from other commonly accepted reengineering and management best practices. SSA has already begun to apply some lessons it has learned, such as strengthening executive oversight of its redesign effort. However, the extraordinary difficulty of the task at hand and the performance shortcomings previously experienced suggest that these steps might not be enough. Other fundamental changes in SSA’s approach will probably be necessary. In particular, although more focused than its original plan, SSA’s current redesign plan is still very large in scope and difficult to manage, and the successful completion of key initiatives will likely require that SSA scale back its near-term efforts even further. SSA can also modify its testing approach to avoid pitfalls encountered in the past. As it moves to implement changes that appeared efficacious in a testing environment, SSA can ensure that it has adequate performance measures and goals to assess changes to the process and to provide early warning of problems as well as adequate quality assurance processes to guard against unanticipated results.

Many Steps Remain Under the Revised Plan

The need for SSA to improve its disability claims process continues today. SSA’s large pending workload persists, especially at the hearing level. The pending workload at the hearing level grew from 357,564 in 1993 to about 483,712 in 1997. In addition, the average length of time it took to receive a hearing decision upon appeal also grew in the 1990s—from 238 days in 1993 to 386 in 1997. The dramatic growth in initial applications for disability benefits that contributed to these increases and exacerbated long-standing problems has ended. In fact, in recent years the number of individuals applying for disability benefits has declined, which has in turn helped reduce the 1998 appellate backlog to 384,313 and appellate processing time to 371 days. However, no one knows how long this decline will last. An economic downturn could increase unemployment and drive up demand for disability benefits, and the number of applications, at any time. Moreover, the number of applications for disability benefits can be dramatically affected by court cases and changes in the law, such as the possibility of congressional action to increase the retirement age. Finally, SSA expects claims to increase again beginning in 2000, when the eligibility age for full Social Security retirement benefits changes from 65 to 67 years, and more dramatically by 2010, as the baby boom generation approaches its disability-prone years. Taken together, present and future workloads highlight the continuing pressure on SSA to move expeditiously to improve its disability claims process.

Many steps remain to be taken under the agency’s February 1997 redesign plan. As of October 1998, the agency was continuing to test the AO, SDM, and FPM initiatives. Should SSA decide to implement any of these positions or process changes, it will face innumerable steps ahead. For example, SSA will need to seek changes in the law or develop new regulations for many of the changes it is considering, a time-consuming and multistep process. For some of the initiatives, such as the SDM, SSA will also need to provide for training, facilities, equipment, and various clerical and managerial supports. In some cases, SSA will need to develop plans for implementing changes in phases, such as installing new positions at a small number of sites each month. In addition, SSA must guard against unwanted effects that could result from making changes to one part of the process without adequately addressing their impact upon other parts. For example, should SSA decide to eliminate the reconsideration step, SSA will need to be aware of the possibility of, and take steps to guard against, the development of more backlogs at OHA caused by the speedier movement of cases through the process to that level.

While SSA has made important progress, much remains to be accomplished on two other important near-term initiatives: process unification and quality assurance. For example, under its process unification initiative, SSA intends to review and revise established regulations to develop its planned single presentation of policy—a time-consuming task to which SSA has not yet been able to devote adequate resources. Also under process unification, SSA intends to continue providing systematic ongoing training to adjudicators at all decision-making levels and to continue work on several remaining subinitiatives. Under its quality assurance initiative, SSA still needs to ensure that adequate in-line quality assurance procedures are in place for any changes it makes to the process. SSA is also still trying to reach an agreement on a single “end-of-line” quality review mechanism for the whole disability claims process. Once agreement is reached, SSA will need to test this mechanism. As discussed in chapter 2, developing and testing initiatives can involve a substantial amount of time and effort and require the cooperation of numerous stakeholders.

If SSA continues its redesign effort as planned, the agency has even more matters to contend with for its three longer-term initiatives: the DCM; a simplified methodology for making disability decisions; and the Redesigned Disability System (RDS), SSA’s new disability computer system. All three involve major operational changes and are the furthest from implementation. The DCM combines the duties of SSA field office personnel and state disability examiners and will require legislative changes before it can be implemented. As of October 1998, SSA was still conducting the first of three lengthy test phases that precede full implementation of the DCM.The simplified decision methodology initiative is still in the developmental stage, and much more research needs to be accomplished before SSA can begin laboratory testing. Finally, SSA is experiencing a number of problems with its proposed RDS, a system that is viewed as key to many of the planned process efficiencies. In January 1998, we reported that software development problems had delayed the scheduled implementation of RDS by more than 2 years. Later in 1998, we reported that SSA had experienced problems and delays in its RDS pilot effort initiated in August 1997 to assess the performance, costs, and benefits of RDS. For example, systems officials stated that, using RDS, the reported productivity of claims representatives in the SSA field office dropped. Systems officials also stated that because the RDS software had not performed as anticipated, SSA had engaged an independent contractor to evaluate and recommend options for proceeding with RDS. This effort is expected to further delay SSA’s national rollout of the new disability computer system. See table 3.1 for the steps that have yet to be taken under the revised plan.

Even if SSA successfully tests and implements all of the redesign initiatives included in the February 1997 updated plan, it is unlikely that all of the problems that gave rise to SSA’s redesign effort in the first place will be satisfactorily resolved. As we have noted, test results to date show only modest improvements in operations, and budgetary savings will not be as large as originally anticipated. Moreover, except for the AO initiative, most of SSA’s redesign efforts to date are focused on improving the process at the initial determination level, leaving problems at the ALJ level largely unresolved. These problems include length of processing times and the large number of backlogged cases at hearing offices, which are among the most pressing problems that SSA faces with the claims process and which require additional solutions.

SSA’s Actions to Revise Its Approach Have Been Helpful but Insufficient to Ensure Success

SSA has already taken actions to revise its approach and apply some lessons learned from its early efforts with redesign, including formalizing high-level executive oversight, working to improve test design, and rethinking its strategy for communicating with stakeholders. However, these efforts may not be enough to ensure success. Because of the unique barriers to change inherent in governmental operations, redesign is particularly challenging for government agencies, and SSA may need to consider additional changes in its approach to improve its chances of making tangible future progress.

In its 1997 plan, SSA established a new management structure to oversee redesign efforts in order to make its senior managers more accountable and involved. Specifically, SSA centralized authority for redesign efforts by creating an Executive Committee for Disability Redesign, chaired by the principal deputy commissioner. Such high-level oversight is critical, given the organizational complexity of the disability claims process. It is also consistent with government and industry best practices, which provide that the individual in charge of a reengineering effort be responsible for the entire process and its performance. Strengthening executive oversight has already had a positive effect on the progress of redesign. For example, by promoting timely processing of cases for the FPM test, the Executive Committee has helped to expedite analysis of test results. SSA officials told us they believe that Executive Committee oversight has helped provide a new momentum by working to ensure that activities stay on schedule and that critical policy decisions receive sufficient and early high-level attention.

SSA is also applying some valuable lessons learned from conducting the AO test. Because the AO test results were inconclusive as a result of problems with the design and management of the test, SSA has taken greater care with the design and management of subsequent tests. For example, SSA enlisted the services of an independent consulting firm to review its proposals for both the FPM and DCM tests. Also, for the DCM test, which is similar to the AO test in that it is lengthy and involves testing the efficacy of a new position, SSA is taking steps to ensure test participants receive adequate training and support, and that the testing environment remains stable.

SSA officials told us they have also learned a great deal about balancing the need for open communication with stakeholders with the need to keep initiatives on track and make tough and sometimes unpopular decisions. Effective stakeholder communication is an important area according to reengineering experts. Its importance was recently noted in a private sector survey of 102 private and government organizations that found that sending inconsistent signals and not communicating enough with stakeholders were among the five most serious mistakes top management sponsors made during a major change. However, communicating with stakeholders is different from obtaining consensus on proposed changes, a practice that can sometimes lead to management paralysis. The proposed changes in SSA’s redesign plan affect most aspects of the disability claims process, and it is unlikely that the agency can achieve across-the-board support from all parties affected by the change. Early on in its redesign efforts, SSA leadership took extraordinary steps to reach out to key stakeholders to build acceptance and consensus for its redesign initiatives. SSA officials told us they now understand that they cannot expect to satisfy all stakeholders and believe they lost valuable momentum early in the redesign effort trying to do so. Agency officials have continued with their efforts to communicate with various stakeholder groups, however, and our review showed that, although stakeholders do not unanimously support all of SSA’s redesign initiatives, many of the stakeholders we contacted were satisfied with the level of communication from SSA.

Nevertheless, these positive efforts and lessons learned may prove insufficient for achieving appreciable progress. Even with strengthened executive oversight since February 1997, milestones have continued to slip. Compelling test results and improvements to the disability claims process have also proven elusive. With so much remaining to be accomplished, and many barriers to overcome, SSA will need to take additional steps to keep its redesign effort on track and achieve further improvements to the disability claims process.

SSA is not the only government agency that has had trouble reengineering its operations. According to reengineering experts, many federal, state, and local agencies have failed in their reengineering efforts. One reason for this high degree of failure is the unique environment of the government workplace, which adds considerable complexity. For example, the flexibility to reengineer a process is often constrained by laws or regulations that require that processes follow certain procedures—such as the requirement, in some cases, that a physician participate in disability cases involving children or mental impairments. Also, government agencies, unlike their private sector counterparts, cannot choose their customers and stakeholders. Agencies must serve multiple customers and stakeholders, who often have competing interests. In addition, following government procedures, such as drafting and issuing new regulations and complying with civil service rules, makes it difficult to implement changes at the quick pace often considered vital for successful reengineering efforts. Finally, public agencies must also cope with frequent leadership turnover and changes in the public policy agenda. For example, as discussed in chapter 2, SSA faced several policy changes during the last few years, such as the need to redetermine the eligibility of thousands of children receiving SSI benefits, at the same time that the agency was trying to conduct large tests of process changes.

Sharper Focus Is Needed on Most Critical Areas

According to experts in the field, reengineering requires sharp focus and enormous discipline, and organizations are more likely to succeed if they concentrate their efforts on a small number of initiatives at any given time. One way of focusing a reengineering effort is by prioritizing process improvement objectives and identifying those initiatives most likely to achieve those objectives. Basic reengineering precepts suggest that an agency should decide which process or major subprocess should have highest priority for agency action. This decision should be based on selecting process changes that (1) have strong links to the agency’s mission and would have a high impact on customers, (2) are likely to provide a large return on invested resources, (3) enjoy a strong consensus, (4) are feasible given the available resources and infrastructure, or (5) can be achieved within a short period of time in order to gain experience in reengineering.

SSA’s own experience strongly underscores the need for focus. As discussed in chapter 1, SSA realized early on that it could not effectively manage the large number of initiatives in its original redesign plan within established time frames, and later that scaling back its plan in February 1997 was a step in the right direction. SSA’s experience was not unlike that of others. Early reengineering theory called for large systemwide changes over a short period of time; but experts now suggest that achieving significant change takes longer and costs more than generally believed several years ago. However, SSA has continued to miss milestones and, with much remaining to be accomplished, additional focus may be necessary to achieve significant and concrete improvements to the process. As we reported in December 1996, process unification, quality assurance, and enhanced information systems are among those initiatives most crucial to producing significant improvements in the process. Other initiatives could be explored on a limited basis or undertaken at a later date once progress was ensured for critical initiatives or when additional resources became available.

Concern over the scope of SSA’s plan and the resources used for redesign activities was similarly expressed by the independent, bipartisan Social Security Advisory Board in an August 1998 report. The Board concluded that the costs of the redesign project were significant and could not be sustained indefinitely. The cost of SSA’s redesign efforts is difficult to calculate. According to SSA officials, the agency spent approximately $16.7 million from 1995 through 1998 on redesign activities—mostly on travel associated with relocating test participants around the country, but also on training, rent, supplies, and equipment. In addition to these expenditures, the Advisory Board pointed out that the redesign effort consumed the time and attention of a considerable number of the most experienced and knowledgeable staff within both SSA and the DDSs, diverting them from the routine disability claims process. In the context of constrained administrative resources, the Board advised that resources that had been diverted be returned as soon as possible to their usual functions so that SSA and the state agencies could fulfill their basic program responsibilities.

Prioritizing its key redesign objectives might help SSA to better focus its efforts. As discussed in chapter 1, SSA’s redesign effort currently has five key objectives: allowing claims that should be allowed at the earliest possible level and improving efficiency, speed, and customer and employee satisfaction. However, these objectives can work at cross purposes; an improvement in one area can result in a deterioration of performance in another. For example, focusing on efforts that speed up the process and improve efficiency might reduce the amount of attention given to developing evidence and documenting decisions. This, in turn, might result in incorrect allowances (or denials) earlier in the process. On the other hand, focusing on the objective of making the right decision at the earliest possible level could add time at the initial level, which might result in more accurate initial determinations and fewer appeals, which in turn might improve the speed and efficiency of the overall process.

SSA officials told us that if they were to begin again, they would consider dividing the redesign effort into smaller, more manageable segments. This would be one way for the agency to better focus on specific initiatives and perhaps be able to achieve more visible near-term gains. In fact, SSA may end up taking this approach during the implementation phase by rolling out small segments of the redesign plan one at a time.

A Modified Testing Approach Could More Efficiently Provide Valuable Information

Reengineering best practices, as well as SSA’s own experience to date, suggest that modifications to SSA’s testing approach could help the agency to more efficiently demonstrate the likely result of proposed changes. Conducting smaller and more integrated tests could free up resources to address critical initiatives while effectively demonstrating the efficacy of interrelated changes. In addition, some of SSA’s redesign initiatives face considerable barriers to implementation because they represent significant change, affect jobs, or depend on other changes or supports to be effective. SSA could more effectively explore the viability of such initiatives—as well as of alternative approaches—on a small scale or wait until essential supports have been developed before investing significant resources in testing these initiatives.

Small-Scale Testing Is an Effective Way to Initially Demonstrate Efficacy

Many reengineering experts believe that entities undergoing reengineering, such as SSA, should conduct small tests of proposed initiatives. Reengineering best practices caution against moving directly from concept to large-scale testing or implementation and suggest that methods such as limited pilot tests and prototyping are cost-effective means for evaluating the effectiveness and workability of proposed changes. As we recommended in our 1996 report on SSA’s reengineering effort, SSA would benefit from concentrating its efforts on first testing initiatives using a smaller, more manageable scope at only a few sites across the country.

SSA’s own experience with the AO and SDM tests confirmed that small-scale testing is prudent. Significant resources and time were devoted to large-scale tests of the AO and SDM, only to discover that their efficacy in a stand-alone environment was marginal. The AO test in particular—which lacked good design, disciplined management, and key supports—proved costly and ineffective in proving the AO concept. SSA moved quickly from concept to large-scale testing because it wanted to definitively demonstrate the positive impact of these proposed changes so they could be immediately implemented. But test results did not support immediate implementation. Instead, the outcome has been continual testing that has drained agency resources and energies. In hindsight, SSA could have discovered the marginality of stand-alone initiatives with a much smaller commitment of resources.

As noted in chapter 2, SSA officials continue to believe that the agency must conduct tests involving a large number of cases. Given SSA’s desire to collect a sufficiently large amount of data and move quickly to change the claims process, SSA officials believe their approach for the AO and SDM tests was correct, and that if the test results had been positive, all would be well. However, we believe that SSA took a costly risk that may have eroded support for the initiatives. SSA officials have said that in the future they would consider reducing the number of sites that they use in tests by concentrating test sites in a few states or within one SSA region to permit more efficient use of resources and easier test management and oversight.

SSA’s current plans involve testing other initiatives, such as the DCM, on a large scale. The DCM test currently under way has a start-up cost of $20 million and involves 210 federal and state participants at 33 sites across the country. Given the uncertainties inherent in this new position, as well as SSA’s past experience with large-scale testing of the AO and SDM initiatives, SSA runs the risk of learning on a large and expensive scale that the DCM does not meet the agency’s redesign objectives. It would be more cost-effective to test this initiative on a small scale and move on to a large-scale test only if initial results suggested the potential for significant gains. In the event of unforeseen difficulties or poor test results, it would be easier and less costly to make any necessary adjustments to a small-scale test than to a larger one.

Integrated Testing Is Important for Assessing the Effectiveness of Interrelated Initiatives

According to a reengineering expert we consulted, stand-alone testing of interrelated initiatives is inefficient and unnecessary because it provides no synergy or learning across the whole process. In addition, as shown by reengineering research, effectively evaluating the overall impact of a redesign effort requires studying the entire business unit or process being reengineered. In fact, we recommended in our 1996 report that SSA combine key initiatives into an integrated process and test that process at a few sites.

SSA’s experience confirms the importance of integrated testing. Projected benefits from reengineering were predicated on the assumption that most process changes and supporting initiatives would be operational simultaneously. However, as discussed in chapter 2, SSA has been testing initiatives independently and without the benefit of some key supports. SSA officials maintain that they have learned a great deal from the large-scale, stand-alone tests, such as how to better run a test. They also maintain that the stand-alone tests provided a baseline of information; for example, testing the SDM in a stand-alone environment provided data to compare with the SDM performance in an integrated environment. SSA officials also believe that the tests contributed to improved communication among operational units and opened the door for important cultural changes needed to support redesign. Although SSA may have learned from its stand-alone tests, these tests did not demonstrate dramatic improvements to the process or provide valuable insight on how the AO and SDM would ultimately work in concert with other initiatives. For example, only when SSA began the FPM test did it become apparent that the SDM might have performed differently if it had been tested in an integrated environment. Rather than conducting large-scale testing of individual initiatives, such as the SDM and AO, moving directly into integrated testing, even on a small scale, or waiting until key supports were in place, might have been more efficient.

SSA has the opportunity to apply these lessons in future tests of initiatives. For example, the agency recently began testing expanded rationales —an effort designed to more fully document, at the initial level, the reasons a claim has been denied. These tests have been conducted outside of the FPM test, even though expanded rationales are closely related to other process changes in the FPM. Officials are now taking steps to incorporate this feature into the FPM. SSA will be conducting small pilot tests in four states to gather information regarding the impact of expanded rationales when they are added to other FPM process changes. Folding the expanded rationales test into the FPM test will provide more valuable information on the efficacy of this change in the environment in which it was intended to be implemented. Similarly, a new simplified disability decision methodology and computer software support are considered essential to the success of the DCM position. However, since these important support initiatives are not scheduled to be available in time to meet the current schedule for testing the DCM, it is not clear what or how much SSA will learn from this test about the viability or effectiveness of the DCM in a redesigned environment.

Alternative Redesign Processes Should Be Fully Explored

Reengineering best practices suggest that, before selecting a specific process change for implementation, an organization should develop several possible alternatives to the existing work process and consider the costs and benefits of each. These alternatives should then be explored in order to (1) convincingly demonstrate the potential of each option to achieve the desired performance goals; (2) fully describe the types of technical and organizational changes necessary to support each goal; and (3) if possible, test key assumptions. Also, as part of a cost-benefit analysis, an agency should take into consideration any barriers to and risks in implementing each alternative.

SSA might have avoided some of the problems currently being experienced with the AO initiative, which has engendered strong opposition, had other alternative work subprocesses also been explored on a small-scale basis before large-scale AO testing. Alternatives to the AO initiative for improving the appellate process exist, such as SSA’s temporary program to permit senior staff attorneys in hearing offices to allow benefits in clear-cut cases. However, SSA did not adequately assess the merits of the alternatives by obtaining concrete and comparable data on their relative costs, benefits, and risks. After 3 years of testing, SSA must decide whether to abandon the AO initiative, begin seriously exploring other solutions to pressing problems at the appellate level, or both. Compounding matters, opponents of the AO concept have pointed to its marginal test results to support their own favored, albeit untested, alternatives. SSA officials agreed that they did not fully prepare themselves for the possibility that their proposed changes might not work and thus did not adequately pursue alternatives earlier in the redesign process or develop contingency plans.

SSA may still be able to apply this important lesson in a remaining area by more fully exploring feasible alternatives to the DCM initiative. As with the AO, the DCM initiative is facing some strong opposition and has perhaps even more barriers to full implementation standing in its way. According to one high-level SSA official, test results would have to be very compelling to support implementation of the DCM initiative. Nevertheless, SSA has begun a 3-year large-scale test of the DCM without adequately exploring feasible alternatives. For example, SSA could have—as we recommended in our 1996 report—systematically tested alternatives such as sequential interviewing to compare their relative effects on the process before beginning the large-scale DCM test. Instead of testing this concept, SSA allowed the individual operating units to decide whether or not they would adopt this approach. SSA officials believe few, if any, units are actively pursuing it. There is still time for SSA to explore such alternatives to the DCM while the agency conducts its protracted test.

Implementation Will Require Careful Monitoring

As of October 1998, SSA was considering widespread implementation of several changes to the disability claims process on the basis of some promising results from its FPM test. While SSA has encountered considerable challenges in testing its initiatives, the risk of further difficulty during their implementation is very high. The experience of other public and private organizations that have attempted business process reengineering strongly indicates that, when compared with developing or testing possible changes to a process, implementation of those changes is more difficult. Moreover, it is possible that certain process changes may not perform as expected outside the test environment. SSA, therefore, needs adequate performance goals and measures for key initiatives and objectives to monitor and assess the impact of any changes made to the process. SSA also needs an adequate quality assurance process in place to ensure the quality and accuracy of decisions.

Implementing a New Process Is Difficult

Experience has shown that implementation of a new process is extremely difficult and, compared with development and testing, is the most failure-prone phase of a reengineering effort. During implementation, an organization’s natural resistance to change must be overcome. According to a reengineering expert we consulted, many reengineering efforts fail because too little time and effort are allotted to implementation. The numerous issues that need to be considered and planned for include identifying all tasks, time frames, and needed resources for an orderly transition; structuring the rollout of the new process in a way reasonably suited to the nature of the process and the work and structure of the organization; assigning roles and responsibilities for implementation to the individuals who will do the work of the new process; providing a means for collecting and sharing information on implementation problems and solutions; and providing for close monitoring during implementation.

SSA’s implementation plans issued in 1994 and 1997 do not address many of the above considerations. For example, the plans do not address the key roles, responsibilities, and reporting relationships required by the new process. In our discussions with stakeholders, we found increasing anxiety over the fact that some key organizational decisions related to work space, which unit would be responsible for managing the proposed AO positions, and other infrastructure issues had not yet been made. Nor do the implementation plans address how SSA will monitor the process to ensure successful implementation and optimum improvements. Recognizing that its current implementation plan is lacking in many specifics, SSA plans to develop more detailed implementation plans as key decisions are made.

Adequate Performance Goals and Measures Are Needed to Closely Monitor Results of Process Changes

In order to be able to effectively monitor the results of its process changes during implementation, SSA will need adequate performance goals and measures. Researchers for the Harvard Business Review found that failure to measure a new process can be particularly damaging to a reengineering effort because, without a comprehensive measurement system that can track the new process’ performance, it is impossible to tell if implementation is succeeding or failing. A National Academy of Public Administration report similarly found that measuring and tracking performance continuously was one of six critical success factors in reengineering in the government sector. The report cites performance management as a key characteristic in successful organizations because it offers the only way for them to assess whether or not reengineering is achieving the results they desire.

SSA currently collects a large amount of data related to the disability claims process, but these data could be improved or better tracked for the purpose of determining progress toward redesign goals. Key indicators that SSA uses or could use to measure progress are fragmented, incomplete, or entirely missing. For example, for its agencywide performance plan, SSA is using separate performance measures for disability claims processing times at the initial and appeal levels. This fragmented approach ignores the interrelationship between the two levels; that is, reducing processing time at the initial level might result in premature or poor determinations; cause more cases to be appealed; and, thus, cause overall processing times to increase. Conversely, implementing steps that result in a longer initial processing time but also permit earlier correct allowances could shorten the overall average processing time by reducing appeals. In addition, although SSA has said that process unification is the “cornerstone” in the foundation of the redesigned disability claims process, SSA’s performance plan does not contain a goal for this important initiative; rather, SSA continues to measure performance in a disjointed manner.

SSA is collecting some appropriate data for its tests but still needs to make sure they are linked to the agency’s strategic goals and integrated into the agencywide performance measurement system. As stressed by the Chief Financial Officers Council, an organization composed of representatives of federal departments and agencies, government entities should integrate all reform activities, including reengineering, into the framework of the Government Performance and Results Act (commonly known as the Results Act). According to the Council, one of the reasons this is important is to ensure consistency and reduce duplication of effort. Our review of SSA’s fiscal year 1999 performance plan pointed out that SSA’s reengineering effort is not fully integrated into its Annual Performance Plan. Although the Plan noted SSA’s efforts to improve the disability claims process, the Plan did not include any useful discussion of SSA’s major initiative to completely redesign its disability claims process, nor did it indicate whether changes or improvements expected to result from this effort were factored into the performance measures or goals.

Guarding Against Unintended Results Is Important

SSA cannot be certain that its initiatives will perform the same under “real world” conditions as they did in an artificial test environment, and the agency will need to take additional steps to guard against the possibility of unintended results. For example, SSA’s test of the SDM included a quality review of all cases decided under the test, whereas currently, far fewer cases, most of which involve allowance determinations, are reviewed. In the absence of this 100-percent review, the SDM might perform differently, which could have a significant effect on the accuracy of determinations, the number of allowances and appeals, and overall benefit outlays. Possible unintended results could include inaccurate disability determinations, unanticipated increases in benefit outlays, and increased appellate workloads. When test results are marginal, there is a greater chance that expected process improvements might not materialize.

SSA needs to be sure that, when implementing a change in the process such as the SDM, an adequate quality assurance process is in place to ensure that benefit eligibility decisions are accurate. Accuracy is important because incorrect decisions can result in wrongful benefit payments, unnecessary appeals, or hardship to the claimants caused by incorrect denials. Under its quality assurance initiative, SSA is seeking to build quality into the decision-making process using tools such as training, mentoring, peer review, and feedback. SSA has been exploring approaches to in-line quality assurance as part of its SDM phase II test, allowing individual test sites to set up their own processes. During implementation of the SDM, and in the absence of a uniform approach, SSA will need to take steps to ensure that individual state processes are sufficient to maintain quality.

Ultimately, SSA will need to establish a final quality assurance process that will both identify systemic problems with case decisions and measure the success of SSA’s efforts to build quality into the process. As discussed in chapter 1, current reviews of DDS determinations and ALJ decisions are conducted in isolation from each other. SSA has recently instituted a review of ALJ decisions that will help identify inconsistencies in decision-making between the two levels. However, SSA has yet to develop a single quality review mechanism applicable to both levels. SSA has had particular difficulty getting its initial and appellate decision-making levels to agree on a consistent quality assurance process that cuts across all phases of the decision-making process, including reaching agreement on what constitutes a correct decision.

Conclusions, Recommendations, and Agency Comments and Our Evaluation

Conclusions

More than 4 years after releasing its original redesign blueprint, SSA is still struggling to make significant improvements to its disability claims process. While the agency has made some progress with process unification, SSA has missed many of its redesign milestones, and the results of early tests did not support implementation of specific proposed changes. The agency is still conducting a number of tests, including yet another large, nonintegrated test at numerous sites. Also, top agency officials would like to begin making some implementation decisions about new decisionmaker positions and other proposed changes. With so much left to do, SSA still has a window of opportunity, which will not be open for long, to apply some lessons learned to help the agency achieve important improvements to its disability claims process. (SSA is no longer experiencing a dramatic growth in applications for disability benefits, but the agency can expect applications to increase again as the baby boom generation ages or if the economy suffers a downturn.)

SSA’s ability to learn from past experience will be an important ingredient in the success of future efforts. For example, the size of SSA’s tests and the scope of redesign initiatives slowed SSA’s progress under its original 1994 redesign plan. When the agency revised its redesign plan in 1997 to include fewer initiatives and increased executive oversight, similar problems continued to limit progress. Even this revised plan required the agency to move forward on a number of varied fronts simultaneously, and SSA continued to miss key milestones. Again, the agency may have underestimated the challenges of managing stakeholder input and keeping such an ambitious effort on course. Strong project oversight should continue, but it will probably not be enough to ensure timely progress. Therefore, SSA needs to further focus its efforts by prioritizing its objectives and concentrating its resources on the efforts most likely to achieve those objectives. Such efforts should include those that help to improve consistency in decision-making, ensure accurate results, and achieve large efficiencies through the use of technology.

Past experience has shown that a large-scale test of an individual initiative, while providing an abundance of information on how well that initiative performs in isolation from other changes, does not clearly demonstrate how the initiative would function in a redesigned process and is not the most efficient and effective use of resources. Moreover, while SSA hoped that this testing approach would help gain the support of key stakeholders likely to be affected by the changes, it has not done so. To help free up resources and effectively demonstrate the efficacy of proposed changes, SSA should conduct relatively small tests that integrate several of the proposed changes to the process. Smaller tests will allow SSA to more efficiently identify promising concepts before moving to larger-scale testing or implementation. Integrated testing—testing related concepts together and with key supports in place—will help SSA to demonstrate whether proposed changes will perform as intended under the new process.

SSA’s experience with the AO test has also shown the risks inherent in devoting considerable time and resources to a single unproven approach or change. Results of AO tests have been consistently disappointing, and SSA now finds itself faced with the same long-standing problems the AO was intended to remedy without a tested alternative solution. Therefore, in the future, before investing significant time and resources on any initiative, SSA should explore feasible alternatives for changing the process on a small scale. For example, as we have recommended before, SSA should explore sequential interviewing as a feasible and less risky alternative to the controversial DCM position. Exploring alternatives and conducting small, integrated tests of related initiatives before making large investments are sound reengineering and management practices, the wisdom of which has been underscored by SSA’s experience to date.

Since other organizations have found implementation of process changes to be the most failure-prone phase of a redesign effort, SSA is also likely to encounter numerous pitfalls as it attempts to effect process changes in such a complex environment. As a result, it is especially important for SSA to take action to closely monitor the results of changes it makes to the process and watch for early warnings of problems. It is possible that process changes may not operate as expected outside the test environment. It is also possible that some stakeholders who do not support specific changes may act to undermine their success. If process changes do not operate as expected, the results could include inaccurate decisions, unanticipated program costs, increased appellate workloads, and lack of improvement in service to the claimant. Therefore, SSA should immediately establish a comprehensive set of performance goals and measures—a set that cuts across the whole process and is also linked to SSA’s overall strategic and performance plans—in order to assess and monitor the results of changes to the process.

Finally, SSA’s tests of process changes have provided only limited assurance that these changes would not degrade the quality of disability decisions. Specifically, SSA’s tests included artificial steps, such as a quality review of all test cases, that are not likely to be used outside the test environment. Quality is perhaps the most critical aspect of the decision-making process because each inappropriate disability decision does a disservice to claimants, taxpayers, or both. A wrongful denial burdens the claimant and could result in unnecessary administrative costs if the claimant appeals the decision, whereas a wrongful allowance results in a continuous stream of inappropriate benefit payments. Therefore, as changes are made to the process, SSA should ensure that it has a quality assurance process in place that both promotes and monitors the quality of disability decisions.

Recommendations

As SSA proceeds with further exploration and testing of redesign initiatives and considers implementation options, it should take the following steps to improve the likelihood of making key improvements to the disability claims process: further focus resources on those initiatives, such as process unification, quality assurance, and computer support systems, that offer the greatest potential for achieving SSA’s most critical redesign objectives; test promising concepts at a few sites in an integrated fashion; establish key supports and explore feasible alternatives before committing significant resources toward the testing of specific initiatives, such as the DCM; develop a comprehensive set of performance goals and measures to assess and monitor changes in the disability claims process; and ensure that quality assurance processes are in place that both monitor and promote the quality of disability decisions.

Agency Comments and Our Evaluation

SSA mostly agreed with our report’s observations and the thrust of its recommendations. Specifically, SSA agreed that the tests conducted took longer than anticipated and did not result in the budgetary and operational efficiencies originally hoped for in the 1994 redesign plan. SSA also agreed that it should focus on those areas that will make the greatest contributions to improving the quality and timeliness of decisions. As we have recommended, SSA intends to pursue additional process unification and quality assurance activities. The agency also indicated it will pursue elements of the FPM that will significantly improve customer service. Finally, SSA agreed that systems technology must continue to be an important focus of resources.

SSA took issue with our critique of its testing strategy. SSA believes that stand-alone and FPM testing were both needed to gather data and experience that are essential for making responsible decisions. Moreover, SSA believes that testing at fewer sites would not have provided the required information or allowed the agency to complete the tests in less time. While we understand the agency’s desire to conduct large tests in order to obtain statistically valid results, we continue to believe that exploring the efficacy of initiatives initially on a smaller scale before moving to large-scale testing or implementation would result in a better use of resources. Also, because the various initiatives are interdependent, we believe that integrated testing would provide more complete and useful information on how the initiatives will perform in the new process.

SSA also stated that its current approach to testing the DCM is consistent with our concerns and recommendations, in that it recognizes and builds upon what SSA has learned from previous testing experiences. However, we still have reservations about SSA’s current approach to testing the DCM. First, SSA continues to test this new position on a rather large scale without having explored the position’s potential efficacy through prototyping or limited pilot testing. Second, SSA is testing this initiative without the benefit of the key supports (such as a new simplified disability decision methodology and computer software support) upon which its efficacy relies. Finally, SSA is moving forward with the DCM test without having explored the feasibility of alternative approaches.

While agreeing to focus on certain key initiatives, SSA believes that changes to the decision-making process should precede major computer system changes to enable technological developments to be crafted in the manner most supportive of the new process. Similarly, SSA stated that changes to the decision-making process should precede the development of a new quality assurance process, the purpose of which will be to evaluate the quality of the new process. However, we believe that SSA can make substantial progress toward developing these critical supports before finalizing the process changes. For example, certain key aspects of SSA’s quality assurance initiative—such as ensuring the consistent application of policy across all levels of the process and developing agreement on what constitutes a correct decision—need not rely on final changes to the process of making a decision.

Finally, SSA pointed out, and we agree, that the agency’s monitoring and evaluation systems currently capture a significant amount of data related to the disability claims process. However, as our report indicates, these data are not always translated into comprehensive and complete performance goals and measures that look at the efficiency and effectiveness of the process as a whole. As we asserted in our report, SSA’s use of separate performance measures for disability claims processing times at the initial and appeals levels in its agencywide performance plan ignores the interrelationship between the two levels, thereby reducing the usefulness of the performance measures. We also noted the lack of integration of SSA’s redesign objectives with those found in the agencywide performance plan. We believe SSA can do more to make better use of the large amount of data it collects through a carefully crafted set of performance goals and measures.