Social Security Disability: Improved Processes for Planning and Conducting Demonstrations May Help SSA More Effectively Use Its Demonstration Authority
Published: 2004-11-04T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 1980, the Congress has required the Social Security Administration (SSA) to conduct demonstration projects to test the effectiveness of possible program changes that could encourage individuals to return to work and decrease their dependence on Disability Insurance (DI) benefits. To conduct these demonstrations, the Congress authorized SSA, on a temporary basis, to waive certain DI and Medicare program rules and to use Social Security Trust Funds. The Congress required GAO to review SSA's use of its DI demonstration authority and to make a recommendation as to whether this authority should be made permanent.
section_title: What GAO Found
paragraphs: SSA has not used its demonstration authority to extensively evaluate a wide range of DI policy areas dealing with return to work. Despite being given the authority to assess a broad range of policy alternatives, SSA has, until very recently, focused its demonstration efforts mostly on a relatively narrow set of policy issues--those dealing with the provision of vocational rehabilitation and employment services. SSA's recently proposed or initiated demonstrations have begun to address a broader range of policy issues, such as provisions to reduce, rather than terminate, benefits based on earnings above a certain level. However, the agency has no systematic processes or mechanisms for ensuring that it is adequately identifying and prioritizing those issues that could best be addressed through use of its demonstration authority. For example, the agency has not developed a formal demonstration research agenda explicitly identifying its broad vision for using its DI demonstration authority and explaining how ongoing or proposed demonstration projects support achievement of the agency's goals and objectives. SSA's demonstration projects have had little impact on the agency's and the Congress' consideration of DI policy issues. This is due, in part, to methodological limitations that have prevented SSA from producing project results that are useful for reliably assessing DI policy alternatives. In addition, SSA has not established a formal process for ensuring that its demonstration results are fully considered for potential policy implications. For example, SSA does not maintain a comprehensive record of its demonstration results that could be used to build a body of knowledge for informing policy decisions and planning future research. Furthermore, SSA's reporting of demonstration project results has been insufficient in ensuring that the Congress is fully apprised of these results and their policy implications.

section_title: Letter
paragraphs: From 1982 to 2002, the number of disabled workers receiving benefits  under the Disability Insurance (DI) program doubled from 2.6 million to  5.5 million, while payments quadrupled from about $14.8 billion to  $60 billion. Although technological and medical advances, along with  social changes, have increased the potential for some people with  disabilities to participate in the labor force, less than 1 percent of DI  beneficiaries leave the rolls each year because they have reentered the  workforce. The enactment of various DI work incentives that are intended to encourage beneficiaries to work—and, potentially, to leave the rolls— has also had little discernible impact on beneficiaries’ success in returning  to the workforce. This low rate of return to work, coupled with escalating  growth in DI program enrollment and benefit costs over the past few  decades, threatens to exhaust the DI Trust Fund by 2029. Yet relatively  little is known about the factors that help beneficiaries overcome  employment challenges and disincentives and that inhibit them from achieving an earnings level that leads to self-sufficiency. GAO has  designated federal disability programs as a high-risk area, in part because  of the significant growth in the DI program and the long-standing  challenges in devising effective return-to-work policies.
In response to these challenges, the Congress has, since 1980, required the  Social Security Administration (SSA) to conduct demonstration projects to  test the effectiveness of possible program changes that could encourage  individuals to return to work and decrease their dependence on DI  benefits. In fostering return to work, these demonstrations and the  program changes they test are intended to produce savings in the Trust  Funds or improve DI program administration. To achieve these objectives,  SSA’s DI demonstration authority contains several key features that  provide SSA with a potentially valuable tool for assessing the effectiveness  of policy alternatives. One of these features is SSA’s authority to waive  certain DI and Medicare program rules. For example, when conducting  demonstrations, SSA is permitted to exempt certain beneficiaries from  requirements that workers with disabilities earn below a certain amount to  remain eligible for DI benefits or that they wait 24 months to become  eligible for Medicare benefits. Another key aspect of SSA’s demonstration  authority is the requirement that DI demonstration projects be of sufficient  scope and conducted on a wide enough scale to ensure a thorough  evaluation and results that are applicable to the DI program as a whole. In  addition, the legislation authorized SSA to use DI Trust Fund and Old-Age  and Survivors Insurance Trust Fund monies to pay for the demonstrations  and required SSA to periodically report to the Congress on its  demonstration activities, providing, when appropriate, recommendations  for legislative or administrative changes. The Congress initially granted  SSA this demonstration authority for a 5-year period but subsequently  renewed the authority several times, most recently extending it through  2005.
The Ticket to Work and Work Incentives Improvement Act (P.L. 106-170)  mandated that GAO review SSA’s use of its DI demonstration authority  and provide a recommendation as to whether SSA should be given this  authority permanently. To develop a basis on which to make this  recommendation, we examined the extent to which (1) SSA has used the  demonstration authority to test a range of program changes and (2) SSA’s  demonstration authority has been used to inform policy decisions. In  conducting our examination, we reviewed relevant federal statutes and  regulations governing DI demonstration activities and interviewed SSA  officials with current and prior responsibility for demonstration projects.  We also reviewed documents related to SSA’s planning, implementation,  and evaluation of demonstration projects, including agency reports to the  Congress, public notices, and reports analyzing project results. In addition,  we examined other reviews of SSA’s demonstrations (including prior GAO  and Inspector General reports) and interviewed officials from disability  research, advisory, and advocacy organizations. Furthermore, we obtained  information on research and demonstrations conducted by other federal  agencies. We performed our work in accordance with generally accepted  government auditing standards between October 2003 and August 2004.  For more details about our scope and methodology, see appendix I.
subsections: section_title: Background
paragraphs: Established in 1956, DI is an insurance program that provides monthly  cash benefits to workers who are unable to work because of severe long­ term disability. Workers who have worked long enough and recently  enough are insured for coverage under the DI program. To meet the  definition of disability under the DI program, an individual must have a  medically determinable physical or mental impairment that (1) has lasted  or is expected to last at least 1 year or to result in death and (2) prevents  the individual from engaging in substantial gainful activity (SGA).  Individuals are considered to be engaged in SGA if they have countable  earnings above a certain dollar level. Once a person is on the disability  rolls, benefits continue until (1) the beneficiary dies, (2) the beneficiary  becomes eligible for Social Security retirement benefits at full retirement  age, (3) SSA determines that the beneficiary is no longer eligible for  benefits because his or her earned income exceeds the SGA level, or  (4) SSA decides that the beneficiary’s medical condition has improved to  the point that he or she is no longer considered disabled. In 2002, SSA paid  about $60 billion in DI cash benefits to 5.5 million disabled workers, with  average monthly benefits amounting to $834 per person. In addition to  receiving cash assistance, beneficiaries automatically qualify for Medicare  after 24 months of DI entitlement.
During the 1970s, as the number of disability awards increased  significantly and resulted in substantial cost increases for the DI program,  the Congress became concerned about the growth of the DI program and  program rules that provided disincentives to returning to work. To  encourage DI beneficiaries to return to work—and, potentially, to leave  the benefit rolls—the Congress has, over the years, enacted legislation  providing various work incentives. Such incentives include a trial work  period during which beneficiaries may earn any amount for 9 months  within a 60-month period and still receive full cash and medical benefits  and continued Medicare coverage that allows beneficiaries to maintain  eligibility for Medicare for at least 39 months following a trial work period  as long as medical disability continues.
In an effort to further address these issues, the Congress, in 1980, required  SSA to conduct demonstration projects to evaluate the effectiveness of  policy alternatives that could encourage DI beneficiaries to reenter the  workforce. A key aspect of this demonstration authority is SSA’s ability to  waive DI and Medicare program rules to the extent needed in conducting  these projects. The legislation granting DI demonstration authority also  identified a variety of policy alternatives for SSA to consider testing,  including (1) alternative ways of treating DI beneficiaries’ work-related  activity such as methods allowing for a reduction in benefits based on  earnings and (2) modifications in other rules, such as the trial work period  and Medicare eligibility waiting period, that may serve as obstacles to DI  beneficiaries returning to work. In addition, this legislation identified  several requirements pertaining to the design and evaluation of DI  demonstration projects. In particular, these projects were required to be of  sufficient scope and carried out on a wide enough scale to permit a  thorough evaluation of the policy alternatives studied such that the results  would be generally applicable to the operation of the DI program. The law  additionally required SSA to submit reports to the Congress announcing  the initiation of DI demonstration projects as well as periodic reports  describing the status of these projects and a final report on all projects  carried out under the demonstration authority. SSA was directed to make  recommendations, when appropriate, for legislative or administrative  changes in its reports to the Congress.
Another important aspect of SSA’s DI demonstration authority is that  unlike other SSA research activities, which are funded through  congressional appropriations, these projects can be paid for with DI Trust  Fund and Old-Age and Survivors Insurance Trust Fund monies. Therefore,  SSA is not required to obtain congressional approval for DI demonstration  expenditures, although it is required to receive approval from the Office of  Management and Budget for an annual apportionment of Trust Funds for  these demonstrations.
SSA’s DI demonstration authority has always been granted on a temporary  basis and therefore has been subject to periodic review and renewal by the  Congress. After initially granting this authority for a 5-year period, the  Congress subsequently renewed it several times, in 1986, 1989, 1994, 1999,  and 2004. The renewal of SSA’s authority has sometimes been delayed so  that SSA has, on several occasions, gone without DI demonstration  authority. For example, after its demonstration authority expired in June  1996, SSA was not again granted DI demonstration authority until  December 1999. Most recently, the Congress extended this demonstration  authority through December 2005.
In addition to granting this general DI demonstration authority, the  Congress may enact legislative mandates for SSA to conduct specific DI  demonstration projects. For example, the Ticket to Work and Work  Incentives Improvement Act of 1999 required SSA to conduct a  demonstration to assess the effectiveness of a benefit offset program  under which DI benefits are reduced by $1 for every $2 in earnings (above  a certain level) by a beneficiary. SSA’s authority to conduct this  demonstration is similar in some respects to the authority it has under its  general DI demonstration statute. For instance, the statute allows waiver  of DI and Medicare program provisions to carry out this benefit offset  demonstration. However, some differences exist between the two  authorities. In particular, the benefit offset demonstration authority  provides a more detailed and comprehensive list of demonstration  objectives for SSA to fulfill than does SSA’s general authority. For  example, the benefit offset demonstration authority lists six “matters to be  determined,” which include assessments of project costs; savings to the  Trust Funds; and project effects on employment outcomes such as wages,  occupations, benefits, and hours worked.
Regardless of the authority under which they are carried out,  demonstration projects examining the impact of social programs are  inherently complex and difficult to conduct. Measuring outcomes,  ensuring the consistency and quality of data collected at various sites,  establishing a causal connection between outcomes and program  activities, and separating out the influence of extraneous factors raise  formidable technical and logistical problems. Thus, these projects  generally require a planned study and considerable time and expense.  Adding to these complexities are other administrative or statutory  requirements affecting SSA’s DI demonstrations. For example, SSA’s  policy is that its demonstration projects must not make those who  participate in the project worse off, which could limit the specific types of  policy alternatives the agency can study or the methods used to study such  alternatives.
Although the legislation granting DI demonstration authority does not  prescribe the use of particular methodological approaches, SSA has  repeatedly recognized that the law’s general requirements for  demonstration evaluations require SSA to conduct these projects in a  rigorous manner that provides the agency with a reliable basis for making  policy recommendations. Rigorous methods are required to estimate the  net impact of a tested disability policy option because many other factors,  such as the economy, can influence whether a beneficiary returns to work.  In an August 2002 report to the SSA Commissioner, an SSA advisory panel  stated that it is widely agreed that experimental designs, “when feasible  from operational and budgetary perspectives and when they can be  conducted without serious threats to their validity, are the best  methodology for determining the effects of changes in government  programs.” In addition, SSA officials and other researchers have noted  the advantages of experimental designs in providing policymakers with  more clear-cut results that are less subject to debate than results derived  from other methods. However, when experimental designs are not  feasible or desirable, the use of quasi-experimental designs offers a  reasonably rigorous evaluation alternative that may, under certain  circumstances, offer advantages over experimental designs.
Other factors may also limit the rigor of DI demonstrations, including  insufficient sample sizes, inconsistency in demonstration design or  implementation across multiple project sites, and deficiencies in data  collection. Such design, implementation, and evaluation weaknesses may  hamper SSA’s use of project results as a basis for making policy  recommendations because they limit the agency’s ability to (1) control for  factors external to the demonstration, (2) generalize demonstration results  to a wider population of DI beneficiaries, and (3) isolate the effects of  specific policy interventions from the overall effects produced by a  demonstration.
The Office of Program Development and Research (OPDR) is the entity  within SSA that develops and implements demonstration projects for the  DI and Supplemental Security Income (SSI) Programs. OPDR program  and research staff—sometimes with the assistance of outside research  organizations—identifies the broad outlines and requirements of disability  program demonstration projects, including the basic objectives, scope,  and methodological standards for these projects. SSA then issues formal  notices requesting public or private sector organizations to submit offers  to conduct the demonstration projects, which may include development of  a detailed design plan, provision of technical support, collection of project  data, or evaluation of project results. On the basis of SSA’s review of  submitted proposals and bids, the agency may enter into grants,  cooperative agreements, or contractual arrangements with one or more  organizations to carry out demonstration projects. For example, a single  demonstration may involve cooperative agreements with states to design  and implement projects as well as contracts with one or more research  institutions to provide technical assistance to the states and evaluate  demonstration results. Project managers in OPDR have the primary  responsibility for overseeing demonstration projects to ensure that they  meet SSA’s technical and programmatic requirements. OPDR collaborates  with SSA’s Office of Acquisition and Grants in issuing formal project  notices and solicitations and, subsequently, in overseeing grant or contract  performance.
section_title: SSA Has Not Used Its Demonstration Authority to Evaluate a Wide Range of DI Return-to-Work Policy Issues
paragraphs: SSA has not used its demonstration authority to extensively evaluate a  wide range of DI policy areas dealing with return to work. Until very  recently, SSA has focused its demonstration efforts primarily on a  relatively narrow set of policy issues dealing with the provision of  vocational rehabilitation and employment services, despite being given the  authority to assess a much broader range of policy alternatives. Even in  the area of vocational rehabilitation and employment issues, SSA’s use of  DI demonstration authority has not been comprehensive and, therefore,  did not extensively address key policy issues that the agency is currently  grappling with under its Ticket to Work program. SSA’s recently initiated  or proposed demonstrations have begun to address a broader range of  policy issues. However, the agency has no systematic processes or  mechanisms for ensuring that it is adequately identifying and prioritizing  those issues that could best be addressed through use of its demonstration  authority.
subsections: section_title: SSA Has Not Used Its Demonstration Authority Extensively
paragraphs: The DI demonstration projects that SSA has conducted since 1980 have  not extensively addressed a wide range of return-to-work policy issues.  Since first being granted DI demonstration authority 24 years ago, SSA has  used this authority to complete four projects, with another project nearing  completion.Total costs for these projects amount to at least $107 million,  of which about $42 million was paid for from the Old-Age and Survivors  Insurance and Disability Insurance (OASDI) Trust Funds.
The legislation granting DI demonstration authority to SSA provided the  agency with an opportunity to examine a broad set of return-to-work  policy alternatives and even identified some specific alternatives for SSA  to consider studying, including (1) reducing, rather than terminating,  benefits based on earnings; (2) lengthening the trial work period;  (3) decreasing the 24-month waiting period for Medicare benefits;  (4) altering program administration; (5) earlier referral of beneficiaries for  rehabilitation; and (6) using employers and others to stimulate new forms  of vocational rehabilitation. The projects SSA has conducted thus far have  focused predominantly on the latter category of issues involving  vocational rehabilitation and have focused to a lesser extent—or not at  all—on other key policy issues affecting return to work (see table 1). More  specifically, examination of policy alternatives dealing with the provision  of vocational rehabilitation and employment services has been the primary  objective of four of the five completed or nearly completed  demonstrations. Although two of these projects also examined other DI  return-to-work policy issues—such as the possible effects of changes in  program work incentives and alterations in the provision of medical  benefits—they did so to only a limited extent. None of the projects looked  at other potentially significant DI policy issues, such as the possibility of  changing SSA’s benefit structure to allow for a reduction in benefits,  rather than a complete cutoff of benefits, based on earnings.
Furthermore, SSA has not used its DI demonstration authority to  comprehensively examine issues involving vocational rehabilitation,  including key policy issues with which the agency is currently grappling.  For example, SSA did not extensively test key elements of what eventually  became the Ticket to Work program. Although the ticket program was  not formally proposed by SSA in a legislative package until 1997, as early  as 1989, in an annual report to the Congress on SSA’s demonstration  activities, SSA noted that among its ideas for improving SSA’s ability to  assist beneficiaries in returning to work was a voucher program that could  be used to pay for vocational rehabilitation services from private  providers. SSA told the Congress that such a program, as well as other  possible policy changes, would need to be thoroughly tested as a  prerequisite to developing a new nationwide program. However, only one  project completed under SSA’s DI demonstration authority—Project  Referral System for Vocational Rehabilitation Providers (Project RSVP),  initiated in 1997—addressed an issue directly relevant to the ticket  program, namely, the use of a contractor to perform certain administrative  functions for an expanded vocational rehabilitation referral and  reimbursement program. But our review of project documentation and our  discussions with SSA officials indicate that Project RSVP was more of an  effort to make an operational change in the way SSA managed its  vocational rehabilitation program than a study to evaluate the advantages  and disadvantages of such a change. In fact, we could not identify any end  product or final results for this project.
SSA also made another attempt, ultimately unsuccessful, to directly  address issues related to establishment of a ticket program. In the  Omnibus Budget Reconciliation Act of 1990, the Congress mandated that  SSA use its DI demonstration authority to assess the advantages and  disadvantages of permitting DI beneficiaries to select from among both  public and private vocational rehabilitation providers. But in January  1993, SSA reported to the Congress that it would be unable to conduct this  demonstration because of an insufficient number of providers willing to  participate in the project. SSA explained that the performance-based  reimbursement provisions of the proposed project appeared to be the  reason why providers were reluctant to participate. Despite the Congress’  expressed interest in these issues, SSA did not attempt to identify  alternative ways to carry out such a demonstration. In particular, given  that SSA remained very interested in the expanded use of private  rehabilitation providers for the DI program, the difficulties encountered in  recruiting providers for the demonstration should have highlighted the  need for SSA to further study the issue of provider reimbursement before  proceeding with any policy initiatives in this area. SSA’s current Deputy  Commissioner for Disability and Income Security Programs told us that if  SSA had used its demonstration authority to study these types of issues in  the 1990s, SSA might have been able to identify and possibly resolve these  issues then rather than struggling to do so now. In addition, such  information could have been helpful in the Congress’ consideration of the  ticket legislation’s merits as it deliberated whether to enact this program.
section_title: SSA’s Recently Proposed Demonstrations Address a Broader Range of Policy Issues
paragraphs: In contrast to the completed and nearly completed demonstration  projects, SSA’s more recent projects, which are generally in the early  planning or proposal stages, represent a much more wide-ranging set of  demonstrations (see table 2). For example, the projects, as currently  described, will deal with a variety of issues such as early provision of cash  and medical benefits and a change in the benefit payment structure to  allow a benefit offset for beneficiaries earning above the SGA level. This  more comprehensive approach to demonstrations is due in part to  legislative changes. For example, the Ticket to Work Act mandated that  SSA conduct a benefit offset demonstration and also permitted SSA, for  the first time, to conduct demonstrations involving DI applicants, thereby  allowing SSA to test ideas such as early provision of cash and medical  benefits and vocational rehabilitation services to individuals who have not  yet entered the disability rolls. In addition, SSA has recently placed a high  priority on conducting disability demonstration projects that examine the  key issues affecting beneficiaries’ return to work. This priority was  reflected in the SSA Commissioner’s September 25, 2003, testimony before  the House Committee on Ways and Means, Subcommittee on Social  Security, in which she announced several new demonstrations as part of a  broader strategy to improve the DI and SSI programs. SSA estimates that  these recently proposed and initiated projects will cost about $357 million,  $293 million of which will be paid for from the OASDI Trust Funds.
section_title: SSA Lacks a Formal Process for Establishing a Demonstration Research Agenda
paragraphs: Despite SSA’s recent broadening of the scope of its projects, the agency  does not have in place any systematic processes for identifying and  assessing potential issues that could be well suited for study under SSA’s  demonstration project authority. Therefore, there is no assurance that the  agency will, in future demonstration efforts, maintain its current focus on  a broad array of return-to-work policy issues. Our discussions with SSA  officials and review of a study examining earlier demonstration efforts  indicate that the agency’s agenda for demonstration projects is subject to  significant change over time resulting, in part, from changes in executive  branch and SSA leadership and senior management. The effects of such  changes may include termination of projects or significant delays and  modifications in their planning and implementation. For example, in its  1994 report examining SSA’s Research Demonstration Program, the  agency’s Inspector General noted that changes in SSA leadership had  disrupted the accomplishment of RDP objectives. The disability research  and advisory officials we spoke with also indicated that SSA’s project  priorities and decisions are significantly influenced by larger political and  organizational changes, which may prevent SSA from focusing on long­ term research objectives. One advisory official noted that these difficulties  in long-term planning have occurred despite the fact that the Congress—in  making SSA an independent agency and establishing a 6-year term for the  SSA Commissioner—intended that SSA would be better able to engage in  the type of long-range planning required to address its program needs.
SSA’s approach for identifying and prioritizing demonstrations has varied  through the years. Soon after being granted DI demonstration authority in  1980, SSA developed a detailed demonstration research plan to directly  address the policy issues identified in SSA’s authorizing legislation.  However, our discussions with SSA officials and review of internal agency  documents indicate that the plan was never acted upon because of  competing organizational priorities and concerns over the potential cost of  the demonstrations and possible technical limitations, such as the  adequacy of systems support. Consequently, as its DI demonstration  authority was due to expire in 1985, SSA had not used it to conduct any  demonstrations. In the second half of the 1980s, after its demonstration  authority was renewed, SSA changed course. Partly on the basis of  solicitation of ideas from the public, SSA identified priority areas dealing  mostly with vocational rehabilitation and employment services issues for  which it would issue grants to public and private organizations to conduct  demonstrations. The specific priority areas identified changed from year to  year as SSA attempted to stimulate, test, and coordinate effective  approaches toward employment assistance.
In its required 1991 annual report to the Congress on its DI demonstration  activities, SSA said that it was proceeding with broader testing of key  elements of a comprehensive employment and rehabilitation system. But  our review of agency documents and discussions with SSA officials  indicate that SSA has not developed a formal, comprehensive, long-term  agenda for conducting DI demonstration projects. Senior SSA officials told  us that the agency’s current demonstration project decisions are, to some  extent, based on discussions with outside research, advocacy, and other  groups. But SSA has no formal mechanisms and requirements in place to  ensure that the agency obtains such input and to decide how such input  should be factored in with other considerations in determining the  agency’s demonstration priorities.
The need for explicit planning concerning SSA research, including  demonstrations, has been identified in past reviews of SSA’s disability  programs. For example, in 1998, the Social Security Advisory Board  (SSAB) noted the need for SSA to develop a comprehensive, long-range  research and program evaluation plan for DI and SSI that would guide the  agency’s research and define priorities. SSAB also said that SSA’s  research plan should reflect broad consultation with the Congress, other  agencies, SSAB, and others and recommended the establishment of a  permanent research advisory panel to advise in the development of a long­ range plan. In a 1996 report on SSA’s disability programs, the National  Academy of Social Insurance noted the “dearth of rigorous research on the  disability benefit programs” since the 1980s and said that SSA needed a  comprehensive, long-range research program to address this deficiency.In addition, officials from disability research, advisory, and advocacy  groups told us that they believe the establishment of a formal research  agenda or an advisory panel with regard to demonstration projects would  be helpful in ensuring that SSA adequately identifies its demonstration  priorities and maintains its commitment to these priorities even in the face  of political or administrative changes.
section_title: SSA’s Demonstration Projects Have Had Little Influence on Consideration of DI Policy Changes
paragraphs: SSA’s demonstration projects have had little influence on the agency’s and  the Congress’ consideration of DI policy issues. This is due, in part, to  methodological limitations that have prevented SSA from producing  project results that are useful for reliably assessing DI policy alternatives.  In addition, SSA lacks a formal process for fully considering the potential  policy implications of its demonstration results. Furthermore, SSA’s  reports on demonstration projects have not fully apprised the Congress of  project results and their policy implications.
subsections: section_title: Limitations in Project Design, Implementation, and Evaluation Lessen Usefulness of Demonstration Results
paragraphs: The demonstration projects SSA has conducted under its DI  demonstration authority have generally not been designed, implemented,  or evaluated in a rigorous enough manner to allow the agency to reliably  assess the advantages and disadvantages of specific policy alternatives.  While SSA’s major DI demonstrations have varied significantly in their methodological rigor, all of them have experienced at least some  significant methodological limitations. For example, SSA’s first major DI  demonstration, the Research Demonstration Program, was characterized  by a number of fundamental design and evaluation flaws such as the  limited scope and small sample sizes of the RDP projects and the limited  use of control groups. In its 1994 report on the RDP, the Department of  Health and Human Services’ (HHS) Inspector General noted that because  of such limitations, “grantees were unable to conduct research that SSA  deemed necessary for definitive tests of alternatives to help beneficiaries  obtain work.” In addition, SSA did not develop a plan for evaluating the  overall RDP results as part of its initial project design. In a required  1994 annual report to the Congress on its demonstration activities, SSA  acknowledged that the lack of a rigorous project design and the omission  of a strong evaluation component limited the ways in which the project  results could be generalized. But SSA also described a number of  “observations” that resulted from the RDP and noted that this project had  helped to identify the agency’s future demonstration priorities. However,  given the significant limitations of the RDP, it is unlikely that its results  could have provided a reliable basis for effectively establishing such  priorities.
In its next major DI demonstration effort, Project Network, which was  initiated as the RDP projects were being completed, SSA avoided many of  the major shortcomings of the RDP. For example, Project Network was  rigorously designed, using an experimental approach based on the random  assignment of beneficiaries to treatment and control groups. As a result,  this project produced some reasonably clear results, which SSA  thoroughly evaluated in an effort to assess the overall impact of the tested  policy alternatives. Despite its generally rigorous design, Project Network  also had some limitations that may have, to some extent, limited its  usefulness for policy consideration. For example, in examining the effects  of a case management approach for providing vocational rehabilitation  services, Project Network used four different service delivery models.Although the Project Network evaluation provided information on the  overall effects of a case management approach, it did not provide a basis  for reliably assessing and comparing the separate effects of the four  models even though such an assessment may have provided useful  information for policy considerations. In addition, Project Network did not  produce results that could be generalized to the larger population of  beneficiaries, which, in turn, limited SSA’s ability to assess whether the  tested policy should be implemented on a nationwide basis.
As was the case with Project Network, SSA has made a significant effort  under its State Partnership Initiative demonstration to avoid some of the  problems encountered under the RDP. For example, SSA contracted with  two research institutions to design an evaluation plan for the  demonstration and to provide assistance with technical issues and data  collection to the various states conducting this demonstration. Our  discussions with SSA and contractor officials who have been involved in  this demonstration as well as our own review of SPI project documents  indicate that the efforts of the contractors appear to have introduced a  certain degree of rigor in the design, implementation, and, potentially,  evaluation of this demonstration. For example, SSA’s contractors have  indicated that the SPI “core evaluation” will likely produce useful results  regarding the effects on beneficiary employment of the overall package of  policy alternatives tested under the demonstration. But despite these  efforts, the SPI design also has a number of limitations that could  substantially reduce the usefulness of its results for evaluating the effects  of the demonstration’s individual policy alternatives. For example, SSA  gave each of the 12 participating states significant discretion in designing  and conducting projects, which resulted in 12 distinct state projects. Each  project tested different combinations of policy alternatives, applied  different research methods to study these alternatives, and used varying  approaches to select beneficiaries for participation in the project. SSA  officials told us that such differences across projects make it unlikely that  SPI will produce final results that allow for reliable evaluations of specific  policy alternatives on a national level. SSA and one of its SPI contractors  have also noted other potential limitations in the design and  implementation of SPI, such as problems with the quality of states’ data  collection, that may detract from SSA’s ability to evaluate specific policy  alternatives.
SSA officials currently responsible for planning and conducting DI  demonstrations acknowledged that the agency’s past demonstrations have  generally not provided useful information for policy making largely  because of the limited rigor with which these projects were conducted.  However, they emphasized that the agency has, over the past couple of  years, placed a new emphasis on ensuring that DI demonstrations are  rigorously designed so that the results can be used to effectively evaluate  specific policy options and develop recommendations. In particular, the  officials noted the importance of using, whenever feasible, an  experimental approach in its demonstration projects and of ensuring that  demonstration results can be generalized to the larger population of DI  beneficiaries. The officials also emphasized the need for SSA to hire  additional staff with the expertise needed to carry out methodologically  rigorous demonstration projects.
Aside from the SPI demonstration, all of SSA’s other current DI  demonstrations are in the early design phase or have been proposed only  recently. Therefore, we were not able to assess the methodological rigor of  these projects. However, our review of SSA’s request for proposal (RFP)  for its Benefit Offset demonstration indicates that SSA is making a serious  effort to comprehensively and rigorously study this policy issue. For  example, SSA has proposed using an experimental design with random  assignment to treatment and control groups. Nevertheless, the scope and  complexity of SSA’s proposal suggest that this will be a very challenging  project for SSA to carry out successfully, and that the agency will need to  ensure that its project design avoids some of the pitfalls that have limited  the usefulness of past demonstrations, such as insufficient sample size and  lack of uniformity in tested interventions across sites.
section_title: SSA Does Not Ensure That Project Results Are Adequately Considered and Communicated
paragraphs: SSA does not have procedures or processes in place to ensure that project  results—regardless of any limitations that they may have—are fully  considered by senior officials within the agency for their policy  implications or their implications for future SSA research and  demonstrations. Without such processes, projects that begin with the  support of senior managers under one administration may not receive  adequate attention from a new group of senior managers under a future  administration. Our discussions with current and former SSA officials and  with officials from disability research, advocacy, and advisory  organizations indicate that such shifting priorities have been the norm for  SSA’s DI demonstration projects. For example, several of these officials  told us that when Project Network was completed in 1999, its results were  not formally reviewed and considered by senior SSA managers, in part  because of the changes in presidential administrations and in senior  agency leadership that had occurred since the start of the project. Officials  from one of the groups we spoke with told us that SSA’s consideration of  project results could be improved by the establishment of a panel to  review project results and explore their policy implications.
An additional factor that could limit SSA’s consideration of demonstration  results is the lack of an adequate historical record—reflecting the  outcomes and the problems or issues encountered—of the various  projects that the agency has conducted under its demonstration  authority. SSA has not maintained a formal record of its disability  demonstration project activities and results, so basic information on these  projects—such as project notices, design documents, and evaluation  documents—is in some cases no longer available. As a result, information  on some projects can be obtained only by relying on the recollection of  SSA employees who were around when the study was conducted. While  formal document retention requirements may not dictate that SSA  maintain such information, several SSA officials told us that the agency  would benefit from an institutional record of demonstration activity.  According to these officials, such a record would constitute a body of  knowledge that the agency should be building to improve DI return-to­ work policies. This becomes even more important in light of the expected  retirement of a large percentage of SSA staff during this decade.
In addition to having shortcomings in its consideration of demonstration  results, SSA has not sufficiently communicated the status and results of its  demonstration projects to the Congress. Although SSA had been required  to issue various reports to the Congress regarding its DI demonstration  projects, it has not always produced such reports. For example, although  SSA was required to submit final reports on the use of its demonstration  authority in 1985, 1990, 1993, and 1996, the only final report that SSA  submitted was in 1996. In addition, SSA did not submit annual reports on  its demonstration activities in 7 of the 16 years in which these reports  were required. Furthermore, when these reports have been produced, they  have not provided all of the information needed to fully inform the  Congress of demonstration activities and results. For example, our review  of these reports indicates that they have frequently lacked key information  such as a discussion of a project’s potential policy implications, its  limitations, and the costs of conducting the project.
section_title: Conclusions
paragraphs: In allowing SSA to waive program provisions and use OASDI Trust Fund  dollars, SSA’s DI demonstration authority provides the agency with a  special, and potentially very valuable, means of studying policy  alternatives to improve the agency’s return-to-work programs. SSA has  spent tens of millions of dollars from the OASDI Trust Funds to conduct  these projects—in addition to tens of millions of dollars from SSA’s  general appropriations—and expects to spend hundreds of millions more  within the next 10 years. While these amounts may be small as a  percentage of the total Trust Funds, they nevertheless represent a  substantial use of increasingly limited federal resources. After having this  authority for more than two decades, SSA has yet to use it to propose or  assess major policy options that could result in savings to the Trust Funds.  Because SSA’s use of its DI demonstration authority has yet to achieve the  Congress’ intended results—and because SSA is permitted to draw on  increasingly limited Trust Funds to conduct these demonstrations—we  believe it is important for the Congress to maintain close oversight of  SSA’s use of this authority. We also believe that such oversight would be a  greater challenge if the Congress were to grant this demonstration  authority on a permanent basis.
As the DI Trust Fund approaches exhaustion, the need for programmatic  improvements becomes greater and greater. As part of a broader effort to  address this need, SSA has recently initiated or proposed a number of DI  demonstration projects that, according to SSA officials, are geared toward  producing useful and methodologically sound results. Such results could  provide an important basis for SSA to address some of the long-standing  issues that have led GAO to identify federal disability programs as a high­ risk area. However, the challenges SSA has historically faced in  conducting demonstration projects and the potential for changing  priorities to adversely affect long-range research plans suggests that, in the  long run, SSA may be unable to fulfill these demonstration goals. This is  especially likely if SSA continues its informal approach to prioritizing and  planning demonstrations and assessing their results. Without more formal  mechanisms for establishing its commitment to effective and thorough DI  demonstrations—including the submission of regular reports to the  Congress on the results and implications of its demonstration projects—  SSA will be unable to ensure that the extensive amount of time, effort, and  funding devoted to these demonstrations is well spent.
section_title: Recommendations
paragraphs: To help ensure the effectiveness of SSA’s DI demonstration projects, we  recommend that the Commissioner of Social Security take the following  actions:
Develop a formal agenda reflecting the agency’s long-term plans and  priorities for conducting DI demonstration projects. In establishing this  agenda, SSA should consult broadly with key internal and external  stakeholders, including SSA advisory groups, disability researchers,  and the Congress.
Establish an expert panel to review and provide regular input on the  design and implementation of demonstration projects from the early  stages of a project through its final evaluation. Such a panel should  include SSA’s key research personnel as well as outside disability  experts and researchers. SSA should establish guidelines to ensure that  its project plans and activities adequately address the issues or  concerns raised by the panel or provide a clear rationale for not  addressing such issues.
Establish formal processes to ensure that, at the conclusion of each  demonstration project, SSA fully considers and assesses the policy  implications of its demonstration results and clearly communicates  SSA’s assessment to the Congress. Such processes should ensure that  SSA consults sufficiently with internal and external experts in its  review of demonstration project results and that SSA issues a report to  the Congress clearly identifying (1) major project outcomes, (2) major  project limitations, (3) total project costs, (4) any policy options or  recommendations, (5) expected costs and benefits of proposed options  or recommendations, and (6) any further research or other actions  needed to clarify or support the project’s results. Another key aspect of  such formal processes should be a requirement that SSA maintain a  comprehensive record of DI demonstration projects. This record would  help SSA in establishing an empirically based body of knowledge  regarding possible return-to-work strategies and in deriving the full  value of its substantial investments in demonstration projects.
section_title: Matters for Congressional Consideration
paragraphs: To facilitate close congressional oversight and provide greater assurance  that SSA will make effective use of its DI demonstration authority, the  Congress should consider the following actions:
Continue to provide DI demonstration authority to SSA on a temporary  basis but allow SSA to complete all projects that have been initiated  prior to expiration of this authority. This would provide SSA with  greater certainty and stability in its efforts to plan and conduct  demonstration projects while preserving the Congress’ ability to  periodically reassess and reconsider SSA’s overall use of DI  demonstration authority.
Require that SSA periodically provide a comprehensive report to the  Congress summarizing the results and policy implications of all of its  DI demonstration projects. The due date for this report could either  coincide with the expiration of SSA’s DI demonstration authority or, if  this authority is made permanent or extended for a period greater than  5 years, be set for every 5 years. Such reports could serve as a basis for  the Congress’ assessment of SSA’s use of its demonstration authority  and its consideration of whether this authority should be renewed.
Establish reporting requirements that more clearly specify what SSA is  expected to communicate to the Congress in its annual reports on DI  demonstrations. Among such requirements could be a description of all  SSA projects that the SSA Commissioner is considering conducting or  is conducting some preliminary work on. For each demonstration  project that the agency is planning or conducting, SSA should provide  clear information on the projects’ specific objectives, potential costs,  key milestone dates (e.g., actual or expected dates for RFP, award of  contracts or grants, start of project operations, completion of  operations, completion of analysis, and final report), potential  obstacles to project completion, and the types of policy alternatives  that SSA might consider pursuing depending on the results of the  demonstration. This would provide the Congress with a more complete  understanding of the direction and progress of SSA in its efforts to  fulfill its DI demonstration requirements.
More clearly specify the methodological and evaluation requirements  for DI demonstrations to better ensure that such projects are designed  in the most rigorous manner possible and that their results are useful  for answering specific policy questions and for making, where  appropriate, well-supported policy recommendations. Such  requirements should not be entirely prescriptive given the need for SSA  to have sufficient flexibility for choosing the right methodological  approach based on the specific circumstances and objectives of a  particular demonstration project. However, the requirements could call  for SSA to choose, to the extent practical and feasible, the most  rigorous methods possible in conducting these demonstrations.  Whatever methods are ultimately selected, SSA should be sure that the  methods used will allow for a reliable assessment of the potential  effect on the DI program of the individual policy alternatives being  studied. Finally, SSA’s legislative requirements could be revised to  include a more explicit list of project objectives—such as assessments  of specific employment outcomes, costs and benefits, and Trust Fund  savings—similar to the language that was included under Sections  302(b)(1) and (b)(2) of the Ticket to Work and Work Incentives  Improvement Act.
section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on a draft of this report, SSA agreed with our  recommendations. SSA agreed that in the past it has not used its  demonstration authority to extensively evaluate DI policy but noted that  its recently initiated or proposed demonstrations will play a vital role in  testing program and policy changes. SSA also agreed that the use of  experts in developing demonstration projects is very useful and  commented that it has used the expertise of particular individuals on an ad  hoc basis and plans to continue to use the advice and recommendations of  experts in the development of future demonstrations. Finally, SSA agreed  that a central source of information regarding the results and policy  implications of disability demonstrations needs to be established and  stated that it planned to fully analyze the results of demonstration projects  to inform DI policy decisions. SSA’s comments appear in appendix II.
Copies of this report are being sent to the Commissioner of SSA,  appropriate congressional committees, and other interested parties. The  report is also available at no charge on GAO’s Web site at  http://www.gao.gov. If you have any questions about this report, please  contact me at (202) 512-7215. Other contacts and staff acknowledgments  are listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: To address the mandated objectives, we reviewed legislation authorizing  the Social Security Administration (SSA) to conduct Disability Insurance  (DI) demonstration projects, Congressional reports related to this  legislation, and SSA regulations governing DI demonstration activities. We  also examined internal SSA memorandums and planning documents  discussing proposals to conduct demonstration projects and the nature,  purpose, requirements, and distinguishing features of SSA’s demonstration  authority. We interviewed a wide range of current and former SSA officials  who have had involvement in or responsibility for conducting disability  program demonstration projects, including officials from the Office of  Disability and Income Security Programs (ODISP) and two offices  operating under ODISP—the Office of Program Development and  Research and the Office of Employment Support Programs—as well as  officials from the Office of the Chief Actuary, the Office of Acquisition and  Grants, the Office of Budget, the Office of Strategic Management, and the  Office of Research, Evaluation, and Statistics. We also interviewed  officials from disability research, advisory, and advocacy organizations. In  addition, we examined other reviews of SSA’s disability demonstration  and research programs, including prior GAO and Inspector General  reports and reports from disability research and advisory groups. We also  reviewed SSA budget documents identifying agency spending on disability  program demonstrations and SSA testimony describing agency priorities  related to the DI program in general and demonstration projects in  particular. In addition, we examined SSA’s strategic plan, annual  performance plans, and annual accountability reports.
To obtain detailed information on SSA’s DI demonstration projects, we  reviewed various documents related to SSA’s design, implementation, and  evaluation of demonstration projects including agency reports to the  Congress; public notifications of demonstration projects issued in the  Federal Register; contract, grant, and cooperative agreement solicitation  and award notices issued in the Federal Register or in the Commerce  Business Daily; and project reports submitted to SSA by grantees or  contractors, including project design and evaluation documents. We used  information from these sources to identify key characteristics and  outcomes of each project, including its broad goals, specific study  objectives, types of program waivers applied, methodology, actual or  expected costs, funding sources, major project milestones including actual  or expected initiation and completion dates, project duration, involvement  of outside contractors and grantees, key project strengths and limitations,  and final project results, including any recommendations that may have  been made. The type and extent of information we obtained for each  demonstration project varied widely, in large part because SSA has not  maintained comprehensive documentation on its prior demonstrations. In  addition, documentation on SSA’s more recent demonstrations was very  limited given that these projects are in the early planning and design  stages.
To provide a broader context for understanding SSA’s use of its  demonstration authority, we reviewed other federal agencies’ legislative  authorities for conducting demonstration and research activities. We also  examined reports from GAO and other organizations that evaluated  demonstration and research projects conducted by other federal agencies  or that identified key evaluation and methodological issues related to such  projects.
We performed our work at SSA headquarters in Baltimore, Maryland, and  at various locations in Washington, D.C. We conducted our work between  October 2003 and August 2004 in accordance with generally accepted  government auditing standards.
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: The following individuals also made important contributions to this report:  Jacquelyn D. Stewart, Erin M. Godtland, Corinna A. Nicolaou, Daniel A.  Schwimer, Ronald La Due Lake, Michele C. Fejfar.