Social Security Disability: Improved Processes for Planning and Conducting Demonstrations May Help SSA More Effectively Use Its Demonstration Authority
Released: 2004-11-04T00:00:00

Letter

Background

Established in 1956, DI is an insurance program that provides monthly cash benefits to workers who are unable to work because of severe long­ term disability. Workers who have worked long enough and recently enough are insured for coverage under the DI program. To meet the definition of disability under the DI program, an individual must have a medically determinable physical or mental impairment that (1) has lasted or is expected to last at least 1 year or to result in death and (2) prevents the individual from engaging in substantial gainful activity (SGA). Individuals are considered to be engaged in SGA if they have countable earnings above a certain dollar level. Once a person is on the disability rolls, benefits continue until (1) the beneficiary dies, (2) the beneficiary becomes eligible for Social Security retirement benefits at full retirement age, (3) SSA determines that the beneficiary is no longer eligible for benefits because his or her earned income exceeds the SGA level, or (4) SSA decides that the beneficiary’s medical condition has improved to the point that he or she is no longer considered disabled. In 2002, SSA paid about $60 billion in DI cash benefits to 5.5 million disabled workers, with average monthly benefits amounting to $834 per person. In addition to receiving cash assistance, beneficiaries automatically qualify for Medicare after 24 months of DI entitlement.

During the 1970s, as the number of disability awards increased significantly and resulted in substantial cost increases for the DI program, the Congress became concerned about the growth of the DI program and program rules that provided disincentives to returning to work. To encourage DI beneficiaries to return to work—and, potentially, to leave the benefit rolls—the Congress has, over the years, enacted legislation providing various work incentives. Such incentives include a trial work period during which beneficiaries may earn any amount for 9 months within a 60-month period and still receive full cash and medical benefits and continued Medicare coverage that allows beneficiaries to maintain eligibility for Medicare for at least 39 months following a trial work period as long as medical disability continues.

In an effort to further address these issues, the Congress, in 1980, required SSA to conduct demonstration projects to evaluate the effectiveness of policy alternatives that could encourage DI beneficiaries to reenter the workforce. A key aspect of this demonstration authority is SSA’s ability to waive DI and Medicare program rules to the extent needed in conducting these projects. The legislation granting DI demonstration authority also identified a variety of policy alternatives for SSA to consider testing, including (1) alternative ways of treating DI beneficiaries’ work-related activity such as methods allowing for a reduction in benefits based on earnings and (2) modifications in other rules, such as the trial work period and Medicare eligibility waiting period, that may serve as obstacles to DI beneficiaries returning to work. In addition, this legislation identified several requirements pertaining to the design and evaluation of DI demonstration projects. In particular, these projects were required to be of sufficient scope and carried out on a wide enough scale to permit a thorough evaluation of the policy alternatives studied such that the results would be generally applicable to the operation of the DI program. The law additionally required SSA to submit reports to the Congress announcing the initiation of DI demonstration projects as well as periodic reports describing the status of these projects and a final report on all projects carried out under the demonstration authority. SSA was directed to make recommendations, when appropriate, for legislative or administrative changes in its reports to the Congress.

Another important aspect of SSA’s DI demonstration authority is that unlike other SSA research activities, which are funded through congressional appropriations, these projects can be paid for with DI Trust Fund and Old-Age and Survivors Insurance Trust Fund monies. Therefore, SSA is not required to obtain congressional approval for DI demonstration expenditures, although it is required to receive approval from the Office of Management and Budget for an annual apportionment of Trust Funds for these demonstrations.

SSA’s DI demonstration authority has always been granted on a temporary basis and therefore has been subject to periodic review and renewal by the Congress. After initially granting this authority for a 5-year period, the Congress subsequently renewed it several times, in 1986, 1989, 1994, 1999, and 2004. The renewal of SSA’s authority has sometimes been delayed so that SSA has, on several occasions, gone without DI demonstration authority. For example, after its demonstration authority expired in June 1996, SSA was not again granted DI demonstration authority until December 1999. Most recently, the Congress extended this demonstration authority through December 2005.

In addition to granting this general DI demonstration authority, the Congress may enact legislative mandates for SSA to conduct specific DI demonstration projects. For example, the Ticket to Work and Work Incentives Improvement Act of 1999 required SSA to conduct a demonstration to assess the effectiveness of a benefit offset program under which DI benefits are reduced by $1 for every $2 in earnings (above a certain level) by a beneficiary. SSA’s authority to conduct this demonstration is similar in some respects to the authority it has under its general DI demonstration statute. For instance, the statute allows waiver of DI and Medicare program provisions to carry out this benefit offset demonstration. However, some differences exist between the two authorities. In particular, the benefit offset demonstration authority provides a more detailed and comprehensive list of demonstration objectives for SSA to fulfill than does SSA’s general authority. For example, the benefit offset demonstration authority lists six “matters to be determined,” which include assessments of project costs; savings to the Trust Funds; and project effects on employment outcomes such as wages, occupations, benefits, and hours worked.

Regardless of the authority under which they are carried out, demonstration projects examining the impact of social programs are inherently complex and difficult to conduct. Measuring outcomes, ensuring the consistency and quality of data collected at various sites, establishing a causal connection between outcomes and program activities, and separating out the influence of extraneous factors raise formidable technical and logistical problems. Thus, these projects generally require a planned study and considerable time and expense. Adding to these complexities are other administrative or statutory requirements affecting SSA’s DI demonstrations. For example, SSA’s policy is that its demonstration projects must not make those who participate in the project worse off, which could limit the specific types of policy alternatives the agency can study or the methods used to study such alternatives.

Although the legislation granting DI demonstration authority does not prescribe the use of particular methodological approaches, SSA has repeatedly recognized that the law’s general requirements for demonstration evaluations require SSA to conduct these projects in a rigorous manner that provides the agency with a reliable basis for making policy recommendations. Rigorous methods are required to estimate the net impact of a tested disability policy option because many other factors, such as the economy, can influence whether a beneficiary returns to work. In an August 2002 report to the SSA Commissioner, an SSA advisory panel stated that it is widely agreed that experimental designs, “when feasible from operational and budgetary perspectives and when they can be conducted without serious threats to their validity, are the best methodology for determining the effects of changes in government programs.” In addition, SSA officials and other researchers have noted the advantages of experimental designs in providing policymakers with more clear-cut results that are less subject to debate than results derived from other methods. However, when experimental designs are not feasible or desirable, the use of quasi-experimental designs offers a reasonably rigorous evaluation alternative that may, under certain circumstances, offer advantages over experimental designs.

Other factors may also limit the rigor of DI demonstrations, including insufficient sample sizes, inconsistency in demonstration design or implementation across multiple project sites, and deficiencies in data collection. Such design, implementation, and evaluation weaknesses may hamper SSA’s use of project results as a basis for making policy recommendations because they limit the agency’s ability to (1) control for factors external to the demonstration, (2) generalize demonstration results to a wider population of DI beneficiaries, and (3) isolate the effects of specific policy interventions from the overall effects produced by a demonstration.

The Office of Program Development and Research (OPDR) is the entity within SSA that develops and implements demonstration projects for the DI and Supplemental Security Income (SSI) Programs. OPDR program and research staff—sometimes with the assistance of outside research organizations—identifies the broad outlines and requirements of disability program demonstration projects, including the basic objectives, scope, and methodological standards for these projects. SSA then issues formal notices requesting public or private sector organizations to submit offers to conduct the demonstration projects, which may include development of a detailed design plan, provision of technical support, collection of project data, or evaluation of project results. On the basis of SSA’s review of submitted proposals and bids, the agency may enter into grants, cooperative agreements, or contractual arrangements with one or more organizations to carry out demonstration projects. For example, a single demonstration may involve cooperative agreements with states to design and implement projects as well as contracts with one or more research institutions to provide technical assistance to the states and evaluate demonstration results. Project managers in OPDR have the primary responsibility for overseeing demonstration projects to ensure that they meet SSA’s technical and programmatic requirements. OPDR collaborates with SSA’s Office of Acquisition and Grants in issuing formal project notices and solicitations and, subsequently, in overseeing grant or contract performance.

SSA Has Not Used Its Demonstration Authority to Evaluate a Wide Range of DI Return-to-Work Policy Issues

SSA has not used its demonstration authority to extensively evaluate a wide range of DI policy areas dealing with return to work. Until very recently, SSA has focused its demonstration efforts primarily on a relatively narrow set of policy issues dealing with the provision of vocational rehabilitation and employment services, despite being given the authority to assess a much broader range of policy alternatives. Even in the area of vocational rehabilitation and employment issues, SSA’s use of DI demonstration authority has not been comprehensive and, therefore, did not extensively address key policy issues that the agency is currently grappling with under its Ticket to Work program. SSA’s recently initiated or proposed demonstrations have begun to address a broader range of policy issues. However, the agency has no systematic processes or mechanisms for ensuring that it is adequately identifying and prioritizing those issues that could best be addressed through use of its demonstration authority.

SSA Has Not Used Its Demonstration Authority Extensively

The DI demonstration projects that SSA has conducted since 1980 have not extensively addressed a wide range of return-to-work policy issues. Since first being granted DI demonstration authority 24 years ago, SSA has used this authority to complete four projects, with another project nearing completion.Total costs for these projects amount to at least $107 million, of which about $42 million was paid for from the Old-Age and Survivors Insurance and Disability Insurance (OASDI) Trust Funds.

The legislation granting DI demonstration authority to SSA provided the agency with an opportunity to examine a broad set of return-to-work policy alternatives and even identified some specific alternatives for SSA to consider studying, including (1) reducing, rather than terminating, benefits based on earnings; (2) lengthening the trial work period; (3) decreasing the 24-month waiting period for Medicare benefits; (4) altering program administration; (5) earlier referral of beneficiaries for rehabilitation; and (6) using employers and others to stimulate new forms of vocational rehabilitation. The projects SSA has conducted thus far have focused predominantly on the latter category of issues involving vocational rehabilitation and have focused to a lesser extent—or not at all—on other key policy issues affecting return to work (see table 1). More specifically, examination of policy alternatives dealing with the provision of vocational rehabilitation and employment services has been the primary objective of four of the five completed or nearly completed demonstrations. Although two of these projects also examined other DI return-to-work policy issues—such as the possible effects of changes in program work incentives and alterations in the provision of medical benefits—they did so to only a limited extent. None of the projects looked at other potentially significant DI policy issues, such as the possibility of changing SSA’s benefit structure to allow for a reduction in benefits, rather than a complete cutoff of benefits, based on earnings.

Furthermore, SSA has not used its DI demonstration authority to comprehensively examine issues involving vocational rehabilitation, including key policy issues with which the agency is currently grappling. For example, SSA did not extensively test key elements of what eventually became the Ticket to Work program. Although the ticket program was not formally proposed by SSA in a legislative package until 1997, as early as 1989, in an annual report to the Congress on SSA’s demonstration activities, SSA noted that among its ideas for improving SSA’s ability to assist beneficiaries in returning to work was a voucher program that could be used to pay for vocational rehabilitation services from private providers. SSA told the Congress that such a program, as well as other possible policy changes, would need to be thoroughly tested as a prerequisite to developing a new nationwide program. However, only one project completed under SSA’s DI demonstration authority—Project Referral System for Vocational Rehabilitation Providers (Project RSVP), initiated in 1997—addressed an issue directly relevant to the ticket program, namely, the use of a contractor to perform certain administrative functions for an expanded vocational rehabilitation referral and reimbursement program. But our review of project documentation and our discussions with SSA officials indicate that Project RSVP was more of an effort to make an operational change in the way SSA managed its vocational rehabilitation program than a study to evaluate the advantages and disadvantages of such a change. In fact, we could not identify any end product or final results for this project.

SSA also made another attempt, ultimately unsuccessful, to directly address issues related to establishment of a ticket program. In the Omnibus Budget Reconciliation Act of 1990, the Congress mandated that SSA use its DI demonstration authority to assess the advantages and disadvantages of permitting DI beneficiaries to select from among both public and private vocational rehabilitation providers. But in January 1993, SSA reported to the Congress that it would be unable to conduct this demonstration because of an insufficient number of providers willing to participate in the project. SSA explained that the performance-based reimbursement provisions of the proposed project appeared to be the reason why providers were reluctant to participate. Despite the Congress’ expressed interest in these issues, SSA did not attempt to identify alternative ways to carry out such a demonstration. In particular, given that SSA remained very interested in the expanded use of private rehabilitation providers for the DI program, the difficulties encountered in recruiting providers for the demonstration should have highlighted the need for SSA to further study the issue of provider reimbursement before proceeding with any policy initiatives in this area. SSA’s current Deputy Commissioner for Disability and Income Security Programs told us that if SSA had used its demonstration authority to study these types of issues in the 1990s, SSA might have been able to identify and possibly resolve these issues then rather than struggling to do so now. In addition, such information could have been helpful in the Congress’ consideration of the ticket legislation’s merits as it deliberated whether to enact this program.

SSA’s Recently Proposed Demonstrations Address a Broader Range of Policy Issues

In contrast to the completed and nearly completed demonstration projects, SSA’s more recent projects, which are generally in the early planning or proposal stages, represent a much more wide-ranging set of demonstrations (see table 2). For example, the projects, as currently described, will deal with a variety of issues such as early provision of cash and medical benefits and a change in the benefit payment structure to allow a benefit offset for beneficiaries earning above the SGA level. This more comprehensive approach to demonstrations is due in part to legislative changes. For example, the Ticket to Work Act mandated that SSA conduct a benefit offset demonstration and also permitted SSA, for the first time, to conduct demonstrations involving DI applicants, thereby allowing SSA to test ideas such as early provision of cash and medical benefits and vocational rehabilitation services to individuals who have not yet entered the disability rolls. In addition, SSA has recently placed a high priority on conducting disability demonstration projects that examine the key issues affecting beneficiaries’ return to work. This priority was reflected in the SSA Commissioner’s September 25, 2003, testimony before the House Committee on Ways and Means, Subcommittee on Social Security, in which she announced several new demonstrations as part of a broader strategy to improve the DI and SSI programs. SSA estimates that these recently proposed and initiated projects will cost about $357 million, $293 million of which will be paid for from the OASDI Trust Funds.

SSA Lacks a Formal Process for Establishing a Demonstration Research Agenda

Despite SSA’s recent broadening of the scope of its projects, the agency does not have in place any systematic processes for identifying and assessing potential issues that could be well suited for study under SSA’s demonstration project authority. Therefore, there is no assurance that the agency will, in future demonstration efforts, maintain its current focus on a broad array of return-to-work policy issues. Our discussions with SSA officials and review of a study examining earlier demonstration efforts indicate that the agency’s agenda for demonstration projects is subject to significant change over time resulting, in part, from changes in executive branch and SSA leadership and senior management. The effects of such changes may include termination of projects or significant delays and modifications in their planning and implementation. For example, in its 1994 report examining SSA’s Research Demonstration Program, the agency’s Inspector General noted that changes in SSA leadership had disrupted the accomplishment of RDP objectives. The disability research and advisory officials we spoke with also indicated that SSA’s project priorities and decisions are significantly influenced by larger political and organizational changes, which may prevent SSA from focusing on long­ term research objectives. One advisory official noted that these difficulties in long-term planning have occurred despite the fact that the Congress—in making SSA an independent agency and establishing a 6-year term for the SSA Commissioner—intended that SSA would be better able to engage in the type of long-range planning required to address its program needs.

SSA’s approach for identifying and prioritizing demonstrations has varied through the years. Soon after being granted DI demonstration authority in 1980, SSA developed a detailed demonstration research plan to directly address the policy issues identified in SSA’s authorizing legislation. However, our discussions with SSA officials and review of internal agency documents indicate that the plan was never acted upon because of competing organizational priorities and concerns over the potential cost of the demonstrations and possible technical limitations, such as the adequacy of systems support. Consequently, as its DI demonstration authority was due to expire in 1985, SSA had not used it to conduct any demonstrations. In the second half of the 1980s, after its demonstration authority was renewed, SSA changed course. Partly on the basis of solicitation of ideas from the public, SSA identified priority areas dealing mostly with vocational rehabilitation and employment services issues for which it would issue grants to public and private organizations to conduct demonstrations. The specific priority areas identified changed from year to year as SSA attempted to stimulate, test, and coordinate effective approaches toward employment assistance.

In its required 1991 annual report to the Congress on its DI demonstration activities, SSA said that it was proceeding with broader testing of key elements of a comprehensive employment and rehabilitation system. But our review of agency documents and discussions with SSA officials indicate that SSA has not developed a formal, comprehensive, long-term agenda for conducting DI demonstration projects. Senior SSA officials told us that the agency’s current demonstration project decisions are, to some extent, based on discussions with outside research, advocacy, and other groups. But SSA has no formal mechanisms and requirements in place to ensure that the agency obtains such input and to decide how such input should be factored in with other considerations in determining the agency’s demonstration priorities.

The need for explicit planning concerning SSA research, including demonstrations, has been identified in past reviews of SSA’s disability programs. For example, in 1998, the Social Security Advisory Board (SSAB) noted the need for SSA to develop a comprehensive, long-range research and program evaluation plan for DI and SSI that would guide the agency’s research and define priorities. SSAB also said that SSA’s research plan should reflect broad consultation with the Congress, other agencies, SSAB, and others and recommended the establishment of a permanent research advisory panel to advise in the development of a long­ range plan. In a 1996 report on SSA’s disability programs, the National Academy of Social Insurance noted the “dearth of rigorous research on the disability benefit programs” since the 1980s and said that SSA needed a comprehensive, long-range research program to address this deficiency.In addition, officials from disability research, advisory, and advocacy groups told us that they believe the establishment of a formal research agenda or an advisory panel with regard to demonstration projects would be helpful in ensuring that SSA adequately identifies its demonstration priorities and maintains its commitment to these priorities even in the face of political or administrative changes.

SSA’s Demonstration Projects Have Had Little Influence on Consideration of DI Policy Changes

SSA’s demonstration projects have had little influence on the agency’s and the Congress’ consideration of DI policy issues. This is due, in part, to methodological limitations that have prevented SSA from producing project results that are useful for reliably assessing DI policy alternatives. In addition, SSA lacks a formal process for fully considering the potential policy implications of its demonstration results. Furthermore, SSA’s reports on demonstration projects have not fully apprised the Congress of project results and their policy implications.

Limitations in Project Design, Implementation, and Evaluation Lessen Usefulness of Demonstration Results

The demonstration projects SSA has conducted under its DI demonstration authority have generally not been designed, implemented, or evaluated in a rigorous enough manner to allow the agency to reliably assess the advantages and disadvantages of specific policy alternatives. While SSA’s major DI demonstrations have varied significantly in their methodological rigor, all of them have experienced at least some significant methodological limitations. For example, SSA’s first major DI demonstration, the Research Demonstration Program, was characterized by a number of fundamental design and evaluation flaws such as the limited scope and small sample sizes of the RDP projects and the limited use of control groups. In its 1994 report on the RDP, the Department of Health and Human Services’ (HHS) Inspector General noted that because of such limitations, “grantees were unable to conduct research that SSA deemed necessary for definitive tests of alternatives to help beneficiaries obtain work.” In addition, SSA did not develop a plan for evaluating the overall RDP results as part of its initial project design. In a required 1994 annual report to the Congress on its demonstration activities, SSA acknowledged that the lack of a rigorous project design and the omission of a strong evaluation component limited the ways in which the project results could be generalized. But SSA also described a number of “observations” that resulted from the RDP and noted that this project had helped to identify the agency’s future demonstration priorities. However, given the significant limitations of the RDP, it is unlikely that its results could have provided a reliable basis for effectively establishing such priorities.

In its next major DI demonstration effort, Project Network, which was initiated as the RDP projects were being completed, SSA avoided many of the major shortcomings of the RDP. For example, Project Network was rigorously designed, using an experimental approach based on the random assignment of beneficiaries to treatment and control groups. As a result, this project produced some reasonably clear results, which SSA thoroughly evaluated in an effort to assess the overall impact of the tested policy alternatives. Despite its generally rigorous design, Project Network also had some limitations that may have, to some extent, limited its usefulness for policy consideration. For example, in examining the effects of a case management approach for providing vocational rehabilitation services, Project Network used four different service delivery models.Although the Project Network evaluation provided information on the overall effects of a case management approach, it did not provide a basis for reliably assessing and comparing the separate effects of the four models even though such an assessment may have provided useful information for policy considerations. In addition, Project Network did not produce results that could be generalized to the larger population of beneficiaries, which, in turn, limited SSA’s ability to assess whether the tested policy should be implemented on a nationwide basis.

As was the case with Project Network, SSA has made a significant effort under its State Partnership Initiative demonstration to avoid some of the problems encountered under the RDP. For example, SSA contracted with two research institutions to design an evaluation plan for the demonstration and to provide assistance with technical issues and data collection to the various states conducting this demonstration. Our discussions with SSA and contractor officials who have been involved in this demonstration as well as our own review of SPI project documents indicate that the efforts of the contractors appear to have introduced a certain degree of rigor in the design, implementation, and, potentially, evaluation of this demonstration. For example, SSA’s contractors have indicated that the SPI “core evaluation” will likely produce useful results regarding the effects on beneficiary employment of the overall package of policy alternatives tested under the demonstration. But despite these efforts, the SPI design also has a number of limitations that could substantially reduce the usefulness of its results for evaluating the effects of the demonstration’s individual policy alternatives. For example, SSA gave each of the 12 participating states significant discretion in designing and conducting projects, which resulted in 12 distinct state projects. Each project tested different combinations of policy alternatives, applied different research methods to study these alternatives, and used varying approaches to select beneficiaries for participation in the project. SSA officials told us that such differences across projects make it unlikely that SPI will produce final results that allow for reliable evaluations of specific policy alternatives on a national level. SSA and one of its SPI contractors have also noted other potential limitations in the design and implementation of SPI, such as problems with the quality of states’ data collection, that may detract from SSA’s ability to evaluate specific policy alternatives.

SSA officials currently responsible for planning and conducting DI demonstrations acknowledged that the agency’s past demonstrations have generally not provided useful information for policy making largely because of the limited rigor with which these projects were conducted. However, they emphasized that the agency has, over the past couple of years, placed a new emphasis on ensuring that DI demonstrations are rigorously designed so that the results can be used to effectively evaluate specific policy options and develop recommendations. In particular, the officials noted the importance of using, whenever feasible, an experimental approach in its demonstration projects and of ensuring that demonstration results can be generalized to the larger population of DI beneficiaries. The officials also emphasized the need for SSA to hire additional staff with the expertise needed to carry out methodologically rigorous demonstration projects.

Aside from the SPI demonstration, all of SSA’s other current DI demonstrations are in the early design phase or have been proposed only recently. Therefore, we were not able to assess the methodological rigor of these projects. However, our review of SSA’s request for proposal (RFP) for its Benefit Offset demonstration indicates that SSA is making a serious effort to comprehensively and rigorously study this policy issue. For example, SSA has proposed using an experimental design with random assignment to treatment and control groups. Nevertheless, the scope and complexity of SSA’s proposal suggest that this will be a very challenging project for SSA to carry out successfully, and that the agency will need to ensure that its project design avoids some of the pitfalls that have limited the usefulness of past demonstrations, such as insufficient sample size and lack of uniformity in tested interventions across sites.

SSA Does Not Ensure That Project Results Are Adequately Considered and Communicated

SSA does not have procedures or processes in place to ensure that project results—regardless of any limitations that they may have—are fully considered by senior officials within the agency for their policy implications or their implications for future SSA research and demonstrations. Without such processes, projects that begin with the support of senior managers under one administration may not receive adequate attention from a new group of senior managers under a future administration. Our discussions with current and former SSA officials and with officials from disability research, advocacy, and advisory organizations indicate that such shifting priorities have been the norm for SSA’s DI demonstration projects. For example, several of these officials told us that when Project Network was completed in 1999, its results were not formally reviewed and considered by senior SSA managers, in part because of the changes in presidential administrations and in senior agency leadership that had occurred since the start of the project. Officials from one of the groups we spoke with told us that SSA’s consideration of project results could be improved by the establishment of a panel to review project results and explore their policy implications.

An additional factor that could limit SSA’s consideration of demonstration results is the lack of an adequate historical record—reflecting the outcomes and the problems or issues encountered—of the various projects that the agency has conducted under its demonstration authority. SSA has not maintained a formal record of its disability demonstration project activities and results, so basic information on these projects—such as project notices, design documents, and evaluation documents—is in some cases no longer available. As a result, information on some projects can be obtained only by relying on the recollection of SSA employees who were around when the study was conducted. While formal document retention requirements may not dictate that SSA maintain such information, several SSA officials told us that the agency would benefit from an institutional record of demonstration activity. According to these officials, such a record would constitute a body of knowledge that the agency should be building to improve DI return-to­ work policies. This becomes even more important in light of the expected retirement of a large percentage of SSA staff during this decade.

In addition to having shortcomings in its consideration of demonstration results, SSA has not sufficiently communicated the status and results of its demonstration projects to the Congress. Although SSA had been required to issue various reports to the Congress regarding its DI demonstration projects, it has not always produced such reports. For example, although SSA was required to submit final reports on the use of its demonstration authority in 1985, 1990, 1993, and 1996, the only final report that SSA submitted was in 1996. In addition, SSA did not submit annual reports on its demonstration activities in 7 of the 16 years in which these reports were required. Furthermore, when these reports have been produced, they have not provided all of the information needed to fully inform the Congress of demonstration activities and results. For example, our review of these reports indicates that they have frequently lacked key information such as a discussion of a project’s potential policy implications, its limitations, and the costs of conducting the project.

Conclusions

In allowing SSA to waive program provisions and use OASDI Trust Fund dollars, SSA’s DI demonstration authority provides the agency with a special, and potentially very valuable, means of studying policy alternatives to improve the agency’s return-to-work programs. SSA has spent tens of millions of dollars from the OASDI Trust Funds to conduct these projects—in addition to tens of millions of dollars from SSA’s general appropriations—and expects to spend hundreds of millions more within the next 10 years. While these amounts may be small as a percentage of the total Trust Funds, they nevertheless represent a substantial use of increasingly limited federal resources. After having this authority for more than two decades, SSA has yet to use it to propose or assess major policy options that could result in savings to the Trust Funds. Because SSA’s use of its DI demonstration authority has yet to achieve the Congress’ intended results—and because SSA is permitted to draw on increasingly limited Trust Funds to conduct these demonstrations—we believe it is important for the Congress to maintain close oversight of SSA’s use of this authority. We also believe that such oversight would be a greater challenge if the Congress were to grant this demonstration authority on a permanent basis.

As the DI Trust Fund approaches exhaustion, the need for programmatic improvements becomes greater and greater. As part of a broader effort to address this need, SSA has recently initiated or proposed a number of DI demonstration projects that, according to SSA officials, are geared toward producing useful and methodologically sound results. Such results could provide an important basis for SSA to address some of the long-standing issues that have led GAO to identify federal disability programs as a high­ risk area. However, the challenges SSA has historically faced in conducting demonstration projects and the potential for changing priorities to adversely affect long-range research plans suggests that, in the long run, SSA may be unable to fulfill these demonstration goals. This is especially likely if SSA continues its informal approach to prioritizing and planning demonstrations and assessing their results. Without more formal mechanisms for establishing its commitment to effective and thorough DI demonstrations—including the submission of regular reports to the Congress on the results and implications of its demonstration projects— SSA will be unable to ensure that the extensive amount of time, effort, and funding devoted to these demonstrations is well spent.

Recommendations

To help ensure the effectiveness of SSA’s DI demonstration projects, we recommend that the Commissioner of Social Security take the following actions:

Develop a formal agenda reflecting the agency’s long-term plans and priorities for conducting DI demonstration projects. In establishing this agenda, SSA should consult broadly with key internal and external stakeholders, including SSA advisory groups, disability researchers, and the Congress.

Establish an expert panel to review and provide regular input on the design and implementation of demonstration projects from the early stages of a project through its final evaluation. Such a panel should include SSA’s key research personnel as well as outside disability experts and researchers. SSA should establish guidelines to ensure that its project plans and activities adequately address the issues or concerns raised by the panel or provide a clear rationale for not addressing such issues.

Establish formal processes to ensure that, at the conclusion of each demonstration project, SSA fully considers and assesses the policy implications of its demonstration results and clearly communicates SSA’s assessment to the Congress. Such processes should ensure that SSA consults sufficiently with internal and external experts in its review of demonstration project results and that SSA issues a report to the Congress clearly identifying (1) major project outcomes, (2) major project limitations, (3) total project costs, (4) any policy options or recommendations, (5) expected costs and benefits of proposed options or recommendations, and (6) any further research or other actions needed to clarify or support the project’s results. Another key aspect of such formal processes should be a requirement that SSA maintain a comprehensive record of DI demonstration projects. This record would help SSA in establishing an empirically based body of knowledge regarding possible return-to-work strategies and in deriving the full value of its substantial investments in demonstration projects.

Matters for Congressional Consideration

To facilitate close congressional oversight and provide greater assurance that SSA will make effective use of its DI demonstration authority, the Congress should consider the following actions:

Continue to provide DI demonstration authority to SSA on a temporary basis but allow SSA to complete all projects that have been initiated prior to expiration of this authority. This would provide SSA with greater certainty and stability in its efforts to plan and conduct demonstration projects while preserving the Congress’ ability to periodically reassess and reconsider SSA’s overall use of DI demonstration authority.

Require that SSA periodically provide a comprehensive report to the Congress summarizing the results and policy implications of all of its DI demonstration projects. The due date for this report could either coincide with the expiration of SSA’s DI demonstration authority or, if this authority is made permanent or extended for a period greater than 5 years, be set for every 5 years. Such reports could serve as a basis for the Congress’ assessment of SSA’s use of its demonstration authority and its consideration of whether this authority should be renewed.

Establish reporting requirements that more clearly specify what SSA is expected to communicate to the Congress in its annual reports on DI demonstrations. Among such requirements could be a description of all SSA projects that the SSA Commissioner is considering conducting or is conducting some preliminary work on. For each demonstration project that the agency is planning or conducting, SSA should provide clear information on the projects’ specific objectives, potential costs, key milestone dates (e.g., actual or expected dates for RFP, award of contracts or grants, start of project operations, completion of operations, completion of analysis, and final report), potential obstacles to project completion, and the types of policy alternatives that SSA might consider pursuing depending on the results of the demonstration. This would provide the Congress with a more complete understanding of the direction and progress of SSA in its efforts to fulfill its DI demonstration requirements.

More clearly specify the methodological and evaluation requirements for DI demonstrations to better ensure that such projects are designed in the most rigorous manner possible and that their results are useful for answering specific policy questions and for making, where appropriate, well-supported policy recommendations. Such requirements should not be entirely prescriptive given the need for SSA to have sufficient flexibility for choosing the right methodological approach based on the specific circumstances and objectives of a particular demonstration project. However, the requirements could call for SSA to choose, to the extent practical and feasible, the most rigorous methods possible in conducting these demonstrations. Whatever methods are ultimately selected, SSA should be sure that the methods used will allow for a reliable assessment of the potential effect on the DI program of the individual policy alternatives being studied. Finally, SSA’s legislative requirements could be revised to include a more explicit list of project objectives—such as assessments of specific employment outcomes, costs and benefits, and Trust Fund savings—similar to the language that was included under Sections 302(b)(1) and (b)(2) of the Ticket to Work and Work Incentives Improvement Act.

Agency Comments and Our Evaluation

In commenting on a draft of this report, SSA agreed with our recommendations. SSA agreed that in the past it has not used its demonstration authority to extensively evaluate DI policy but noted that its recently initiated or proposed demonstrations will play a vital role in testing program and policy changes. SSA also agreed that the use of experts in developing demonstration projects is very useful and commented that it has used the expertise of particular individuals on an ad hoc basis and plans to continue to use the advice and recommendations of experts in the development of future demonstrations. Finally, SSA agreed that a central source of information regarding the results and policy implications of disability demonstrations needs to be established and stated that it planned to fully analyze the results of demonstration projects to inform DI policy decisions. SSA’s comments appear in appendix II.

Copies of this report are being sent to the Commissioner of SSA, appropriate congressional committees, and other interested parties. The report is also available at no charge on GAO’s Web site at http://www.gao.gov. If you have any questions about this report, please contact me at (202) 512-7215. Other contacts and staff acknowledgments are listed in appendix III.

Appendix I: Scope and Methodology

To address the mandated objectives, we reviewed legislation authorizing the Social Security Administration (SSA) to conduct Disability Insurance (DI) demonstration projects, Congressional reports related to this legislation, and SSA regulations governing DI demonstration activities. We also examined internal SSA memorandums and planning documents discussing proposals to conduct demonstration projects and the nature, purpose, requirements, and distinguishing features of SSA’s demonstration authority. We interviewed a wide range of current and former SSA officials who have had involvement in or responsibility for conducting disability program demonstration projects, including officials from the Office of Disability and Income Security Programs (ODISP) and two offices operating under ODISP—the Office of Program Development and Research and the Office of Employment Support Programs—as well as officials from the Office of the Chief Actuary, the Office of Acquisition and Grants, the Office of Budget, the Office of Strategic Management, and the Office of Research, Evaluation, and Statistics. We also interviewed officials from disability research, advisory, and advocacy organizations. In addition, we examined other reviews of SSA’s disability demonstration and research programs, including prior GAO and Inspector General reports and reports from disability research and advisory groups. We also reviewed SSA budget documents identifying agency spending on disability program demonstrations and SSA testimony describing agency priorities related to the DI program in general and demonstration projects in particular. In addition, we examined SSA’s strategic plan, annual performance plans, and annual accountability reports.

To obtain detailed information on SSA’s DI demonstration projects, we reviewed various documents related to SSA’s design, implementation, and evaluation of demonstration projects including agency reports to the Congress; public notifications of demonstration projects issued in the Federal Register; contract, grant, and cooperative agreement solicitation and award notices issued in the Federal Register or in the Commerce Business Daily; and project reports submitted to SSA by grantees or contractors, including project design and evaluation documents. We used information from these sources to identify key characteristics and outcomes of each project, including its broad goals, specific study objectives, types of program waivers applied, methodology, actual or expected costs, funding sources, major project milestones including actual or expected initiation and completion dates, project duration, involvement of outside contractors and grantees, key project strengths and limitations, and final project results, including any recommendations that may have been made. The type and extent of information we obtained for each demonstration project varied widely, in large part because SSA has not maintained comprehensive documentation on its prior demonstrations. In addition, documentation on SSA’s more recent demonstrations was very limited given that these projects are in the early planning and design stages.

To provide a broader context for understanding SSA’s use of its demonstration authority, we reviewed other federal agencies’ legislative authorities for conducting demonstration and research activities. We also examined reports from GAO and other organizations that evaluated demonstration and research projects conducted by other federal agencies or that identified key evaluation and methodological issues related to such projects.

We performed our work at SSA headquarters in Baltimore, Maryland, and at various locations in Washington, D.C. We conducted our work between October 2003 and August 2004 in accordance with generally accepted government auditing standards.

Appendix III: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

The following individuals also made important contributions to this report: Jacquelyn D. Stewart, Erin M. Godtland, Corinna A. Nicolaou, Daniel A. Schwimer, Ronald La Due Lake, Michele C. Fejfar.