Social Security Disability: Management Controls Needed to Strengthen Demonstration Projects
Published: 2008-09-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 1980, Congress has required the Social Security Administration (SSA) to conduct demonstration projects to test the effectiveness of possible changes to its Social Security Disability Insurance (DI) and Supplemental Security Income (SSI) programs that could decrease individuals' dependence on benefits or improve program administration. However, in 2004, GAO reported that SSA had not used its demonstration authority effectively. This follow-up report assesses (1) how SSA has used its demonstration authority to test DI and SSI program changes and what information these efforts have yielded and (2) what steps SSA has taken to improve the planning and management of its demonstration projects. To do this, GAO reviewed documents related to SSA's demonstration project management and the steps it took to implement the recommendations in the 2004 report, as well as the projects' designs, evaluations, and costs. GAO also interviewed officials from SSA, its contractors and project sites, and disability experts.
section_title: What GAO Found
paragraphs: Over the last decade, SSA has initiated 14 demonstration projects under its authority to test possible DI and SSI policy and program changes; however, these projects have yielded limited information for influencing program and policy decisions. Of the 14 projects, SSA has completed 4, cancelled 5, and had 5 projects in progress as of June 2008. In total, SSA spent about $155 million on its projects as of April 2008, and officials anticipate spending another $220 million in the coming years on those projects currently under way. Yet, these projects have yielded limited information on the impacts of the program and policy changes they were testing. SSA did not conduct impact evaluations for two of its completed projects, and intended to evaluate five other projects, but could not do so because significant challenges led SSA to cancel them. SSA officials believe the five projects currently under way will yield useful information, but it is too early to tell. SSA has taken steps to improve its demonstration projects but continues to lack management controls to ensure that the projects yield reliable information for making disability policy decisions. SSA has used methodological designs that GAO determined were strong or reasonable when assessed against professional research standards for 11 of its 14 projects. SSA has also used external research professionals to work with the agency on the design, implementation, or evaluation of 12 of the projects, and appointed new program management to oversee its demonstration program. However, as of August 2008, SSA had not fully implemented the recommendations GAO made in 2004 and did not have written policies and procedures governing how it should review and operate its demonstration project program. Specifically, SSA does not have written policies and procedures for its managers and project officers to follow as they design, implement, and evaluate its demonstration projects. Absent such protocols, SSA did not always apply standard research practices, such as conducting pilot phases or obtaining sufficient stakeholder input, which led to data limitations and project cancellations.

section_title: Letter
paragraphs: As of October 2007, the Social Security Administration (SSA) reported that  nearly 8.9 million disabled workers and their dependents were receiving  benefits under the Social Security Disability Insurance (DI) program, while  approximately 7.4 million individuals received federally administered  Supplemental Security Income (SSI) payments. In our prior work, we have  raised concerns about significant growth in the disability rolls and  challenges in devising return-to-work policies. To address these challenges  and encourage individuals capable of working to return to work, Congress  has required SSA to conduct demonstration projects to test the  effectiveness of possible program changes that could decrease individuals’  dependency on benefits. Over the last decade, SSA has spent about $155  million dollars on these projects and anticipates that it will spend another  $220 million on those currently under way.
To conduct these demonstrations, Congress authorized SSA to waive  certain program rules temporarily and use trust fund dollars and  appropriated funds to finance their development. Congress also required  that DI demonstration projects be of sufficient scope and conducted on a  wide enough scale to ensure a thorough evaluation of the program or  policy change under consideration. However, in 2004, we reported that  SSA had not used its DI demonstration authority effectively and that the  demonstration projects conducted in the late 1980s and early 1990s had  little effect on SSA’s and Congress’s consideration of disability policy  issues. We have also placed federal disability programs on our high-risk  list—a list of federal programs that need attention and transformation. We  did so, in part because of concerns about significant growth in the  disability rolls and challenges in devising return-to-work policies. As you  requested, we reviewed the adequacy of SSA’s current and planned  demonstration projects for both DI and SSI programs. Specifically, this  report assesses (1) how SSA has used its demonstration authority to test  DI and SSI program changes and what information these efforts have  yielded and (2) what steps SSA has taken to improve the planning and  management of its demonstration projects.
To complete our work, we reviewed the demonstration projects that SSA  planned, initiated, or terminated between the calendar years 1998 and 2008  in order to identify their key characteristics, including their purpose,  current status, and costs. To gain a better understanding of the  components of SSA’s demonstration projects, we conducted interviews  with staff at sites where SSA implemented 9 of the 14 demonstration  projects. We selected sites that included ongoing, cancelled, and  completed projects, and represented diverse geographic regions  throughout the United States. To determine what information these  projects have yielded about the impacts of the policies and programs being  tested and lessons learned, we collected and reviewed available evaluation  information on each project. We collected data from SSA on the funds  spent to date, projected future expenditures, and total costs for the  demonstration projects, and determined that these data were sufficiently  reliable for the purposes of this report. We also reviewed documents  related to SSA’s planning, implementation, and evaluation of its  demonstration projects, including the development of the projects’  methodological designs. We reviewed the projects’ methodological designs  against professional research standards for demonstration projects. These  research standards were derived from GAO and recognized academic  criteria for conducting evaluation research. Our Office of General Counsel  reviewed these standards and found them consistent with the authorizing  statutes’ methodological requirements. Further, we identified relevant  federal statutes governing DI and SSI demonstration activities and  examined other reviews of SSA demonstration projects that we and SSA’s  Office of the Inspector General had conducted. For both objectives, we  interviewed SSA officials, project officers, and research contractors that  worked on the demonstration projects, officials from SSA’s Office of   Inspector General, and individuals from organizations that have a key role  in federal disability policy.
To address SSA’s planning and management of its demonstration projects,  we interviewed SSA management and staff about the agency’s policies,  guidance and procedures for developing and implementing demonstration  projects, and collected supporting documentation where available. We  assessed the adequacy of SSA’s internal controls using the criteria in  GAO’s Standards for Internal Control in the Federal Government, which  provide the overall framework for establishing and maintaining internal  control in the federal government.
We conducted this performance audit from November 2007 to August 2008  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix I discusses our scope  and methodology in further detail.
subsections: section_title: Background
paragraphs: Under the Social Security Act of 1935, as amended, SSA administers two  federal disability programs—DI and SSI—intended to provide benefits to  individuals with disabilities who are unable to work. The DI insurance  program provides monthly cash benefits to individuals who have a Social  Security work record and the amount of benefits is related to prior  earnings. The DI program is funded primarily through a payroll tax  required by the Federal Insurance Contributions Act (FICA) and is levied  on most workers, and DI benefits are based on an individual’s career  earnings. The SSI program is a means-tested entitlement program that  provides monthly benefits to aged, blind, or disabled individuals who have  very limited income and assets. The SSI program is funded through  general revenues. Unlike the DI benefit, the federal SSI benefit is a flat  amount (adjusted for other income the individual may have) and is not  related to prior earnings.
During the 1970s, as the number of disability awards and costs were  increasing significantly for the DI program, Congress enacted legislation  providing various work incentives to encourage beneficiaries to return to  work and, potentially, leave the benefit rolls. To further these efforts, in  1980, Congress provided SSA with the authority to conduct demonstration  projects to evaluate the effectiveness of policy alternatives that could  encourage both DI and SSI beneficiaries to re-enter the workforce. Under  this authority, SSA can temporarily waive DI and SSI program rules,  including rules regarding program eligibility and benefit administration, in  order to test the effect certain program changes would have on  beneficiaries’ return-to-work rates and the size of the DI and SSI benefit  rolls. Because Congress has historically granted SSA DI demonstration  authority on a temporary basis, it is subject to periodic review and  renewal. Since first providing this authority in 1980, Congress has renewed  it in 1986, 1989, 1994, 1999, and 2004. However, in 2004, Congress only  extended SSA’s DI demonstration authority through December 2005. As a  result, SSA cannot initiate new DI demonstration projects but can  continue those projects that were initiated on or before the December  2005 expiration date. However, SSA can continue to initiate demonstration  projects under its SSI authority. In 2008, SSA requested that Congress  reauthorize its DI demonstration authority, and a bill was introduced to do  so.
SSA’s DI demonstration projects—unlike other SSA research activities— are paid for via the DI trust fund. Therefore, SSA is not required to obtain  congressional approval for DI demonstration expenditures, although it is  required to obtain approval from the Office of Management and Budget for  an annual apportionment of the trust funds for these demonstrations.  Unlike the DI projects, SSI demonstration projects are funded from SSA’s  overall congressional research appropriation. Although SSA’s DI and SSI  demonstration authorities are separate, the agency’s disability  demonstration projects are sometimes jointly authorized when they  involve both DI and SSI beneficiaries and applicants. When a  demonstration project is conducted jointly under the DI and SSI  demonstration authorities, funding for the project is split between trust  fund (i.e., DI) and appropriated (i.e., SSI) sources.
SSA’s Office of Program Development and Research (OPDR) provides  program analysis in support of DI and SSI programs. As part of their  responsibilities, OPDR—sometimes with the assistance of outside  research organizations—identifies the requirements for individual  disability program demonstration projects, including the basic objectives,  scope, and methodological standards for these projects. OPDR project  officers are primarily responsible for overseeing the projects to ensure  that they meet SSA’s technical and programmatic requirements.
As we have previously reported, demonstration projects examining the  impact of social programs aim to provide evidence of the feasibility or  effectiveness of a new approach or practice and are inherently complex  and difficult to conduct. Measuring outcomes, ensuring the consistency  and quality of data collected at various site locations, establishing a causal  connection between outcomes and program activities, and separating out  the influence of extraneous factors can raise formidable technical and  logistical problems. Although the legislation granting SSA its  demonstration authority does not require the use of particular  methodological approaches, SSA has historically recognized that the law’s  general requirement for its demonstration projects requires SSA to  conduct its projects in a rigorous manner that provides the agency with a  reliable basis for making policy recommendations. According to  professional research standards, a rigorous study should include a clearly  stated research question and methodology, including plans for data  collection and evaluation, as well as appropriate controls to determine if a  relationship exists between observed outcomes and the program change  under examination (see app. I).
As part of our prior work related to SSA DI demonstration authority we  reviewed two DI demonstration projects that SSA conducted in the late  1980s and early 1990s. At that time, we found that SSA had not used its  demonstration authority to extensively evaluate a wide range of DI policy  areas dealing with return to work and found that the demonstration  projects had little impact on SSA’s and Congress’s consideration of DI  policy issues. To facilitate close congressional oversight and provide  greater assurance that SSA will make effective use of its authority, we  recommended that SSA develop a formal agenda for its demonstration  projects, establish an expert panel to guide the design and implementation  of its demonstration projects, and establish formal processes to ensure full  consideration of demonstration project results. We also identified several  matters for Congress to consider, including continuation of DI  demonstration authority on a temporary basis, establishment of additional  reporting requirements for demonstrations, and clearer specification of  the methodological and evaluation requirements of demonstrations.
section_title: SSA Has Used Its Demonstration Authority to Initiate Several Projects Since 1998, but Efforts Have Yielded Limited Information
paragraphs: Over the last decade, SSA has initiated 14 demonstration projects to test  policy and program changes, of which SSA has completed 4, cancelled 5,  and had 5 projects in progress as of August 2008. In total, SSA had spent  about $155 million on these demonstration projects as of April 2008, and  officials anticipate spending another $220 million in the coming years on  those projects currently in progress. However, these projects have yielded  limited information for influencing program and policy decisions. We  found that SSA did not conduct impact evaluations for two of its  completed projects and cancelled five projects prior to conducting formal  evaluations; thus, limited information is available.
subsections: section_title: Since 1998, SSA Has Initiated 14 DI and SSI Demonstration Projects in an Effort to Test Policy and Administrative Changes
paragraphs: Since 1998, SSA has initiated 14 projects under its demonstration authority  to test both DI and SSI program changes—6 related to DI, 6 related to SSI,  and 2 examining both programs jointly (see table 1 for an overview of each  project). As of April 2008, SSA spent $80.3 million on its completed  projects, $7.1 million on cancelled projects, and $68.2 million on those  currently in progress.
While SSA initiated 14 projects over the past 10 years, the agency has only  completed 4 of them to date. These completed projects generally focused  on reducing individuals’ dependency on the SSI program by primarily  testing program waivers and other changes in program administration, as  outlined in its SSI demonstration authority. We also found that SSA  cancelled five projects during this period, citing significant challenges that  would have limited the agency’s ability to obtain reliable information from  them.
SSA had five projects in progress as of August 2008. These projects  generally addressed topics outlined in the authorizing legislation for DI  demonstrations and included strategies to return individuals to work and  reduce the growth of certain subgroups of beneficiaries. For example, the  legislation required projects to test various incentives to increase DI  beneficiaries work activity. In addition, the Ticket to Work and Work  Incentives Improvement Act of 1999 provided demonstration authority for  a benefit reduction, rather than complete benefit termination, when  beneficiaries had earnings that exceeded a certain level. To address this  provision, SSA initiated the Benefit Offset National Demonstration  (BOND) shortly after passage of the statute. Another project in progress,  the Mental Health Treatment Study, is focused on identifying strategies for  providing mental health treatment and employment supports for certain DI  beneficiaries with mental illnesses. As of April 2008, officials estimated  that the total costs for the five projects currently in progress would be  about $288 million—about $220 million more than the $68 million already  expended (see table 2).
section_title: Demonstration Projects Have Yielded Limited Information to Influence Program and Policy Changes
paragraphs: Despite using its demonstration authority to examine various issues, SSA’s  demonstration projects have yielded limited information for influencing  program and policy decisions. As required under its demonstration  authority, SSA’s demonstration projects should be conducted in such a  way to permit a thorough evaluation of alternative methods under  consideration. However, we found that SSA had not conducted impact  evaluations—assessments of a project’s effects compared to what would  have happened in its absence—for two of its completed projects, the  Disability Program Navigator and the Florida Freedom Initiative. Thus, no  information about the impacts of the program and policy changes being  tested was available for making decisions about disability policy. The  Disability Program Navigator project, which SSA conducted with the  Department of Labor (DOL), was not evaluated because the evaluation  contractor could not meet SSA’s data security requirements established  after the project was already in progress, and thus could not access the  necessary data. SSA developed a plan to evaluate the Florida Freedom  Initiative after they became concerned about the state’s evaluation plans.  However, SSA did not conduct an evaluation because staff at the state  level conducting the project did not enroll enough participants in the  project to meet sample size requirements. Thus, there was not enough data  available to conduct a reliable evaluation.
Furthermore, SSA intended to evaluate the impacts of policies and  programs being tested in five other projects but could not do so because  the significant challenges those projects faced led SSA to cancel them in  the early stages. Specifically, four of these projects were cancelled prior to  implementation, and thus no data was available to conduct the evaluations  of those policy and programs being tested. The other cancelled project— the Pediatric Medical Unit demonstration—was partially implemented but  not evaluated because the project did not establish the comparison group  needed for the analysis. The project also did not it enroll enough  participants at some implementation sites to meet the sample size  requirements needed to generate data for a reliable evaluation. However,  SSA was able to obtain some preliminary information on how the project’s  strategy appeared to be working at two site locations and is considering  how to use it.
Although SSA did conduct evaluations for two of the completed projects— the Homeless Outreach Projects and Evaluation (HOPE) project and the  State Partnership Initiative project—we found that these projects also  yielded little information about the impacts of the strategies being tested  because the reported evaluation results could not reliably demonstrate the  projects’ effects. For example, an outcome evaluation of the HOPE project  showed that although disability program applicants assisted by the project  received faster decisions from SSA about whether to allow or deny  benefits, another federal agency initiated a similar project even though the  HOPE project was under way. Therefore, SSA’s evaluation results were  weakened, in part because researchers could not separate the effects of  the SSA project from the effects of the other federal project. While SSA did  not obtain reliable impact evaluation results from this project, agency  officials told us that they did obtain a great deal of information about the  process of conducting this type of demonstration project.
For the State Partnership Initiative, we found that SSA did conduct an  impact evaluation when the project ended, but data available at that time  were incomplete, and thus information about the impact of the project  may not be a reliable indicator of the project’s long-term effects. SSA’s  contractors recommended that a final evaluation be conducted once all  the data were collected to assess whether the preliminary results were  valid. However, SSA management chose not to pursue further evaluation  because the preliminary results indicated that the project was not  successful at increasing earnings enough to allow individuals returning to  work to exit the rolls and no longer be dependent on disability benefits.  Nonetheless, SSA’s contractors and agency officials said that lessons  learned from implementing the State Partnership Initiative have influenced  the agency’s subsequent approach to return beneficiaries to work. For  example, SSA used the job descriptions of benefits planners, as well as  data systems from this project, to design the agency’s national Benefits  Planning and Outreach program.
SSA has also begun to obtain some information from one of the five  projects currently under way. SSA has used preliminary results of the  Benefit Offset - 4 State Pilot to aid in the design of the BOND project. Each  of the four states conducting this pilot has provided an interim report to  SSA detailing lessons learned from the implementation of this project.  Because the pilot and BOND both test a benefit offset in conjunction with  other DI program changes, SSA officials and the BOND project contractor  believe that states’ experiences implementing this pilot will help SSA  identify and resolve operational issues before rolling BOND out nationally.  In addition, the four states have conducted preliminary impact evaluations  for the pilot project and expect to complete final evaluations once the  project’s implementation and data collection phases are over.
SSA also plans to conduct impact evaluations of the other demonstration  projects it had in progress as of August 2008. While they have the potential  to yield reliable results, it is too early to tell whether they will ultimately  be useful for informing DI and SSI policy and program changes. These  projects address issues outlined in the demonstration authority statutes  and disability programs more broadly, and SSA officials believe they will  yield useful information. For example, SSA officials anticipate that the  results of the Accelerated Benefits demonstration project could help  policymakers determine whether to eliminate the 24 month waiting period  for Medicare that DI beneficiaries encounter under current law. SSA  officials also anticipate that demonstration projects in progress could yield  key information on how to improve outcomes for certain subgroups of  beneficiaries. For example, SSA officials said that the Youth Transition  Demonstration, which targets young people with disabilities as they  transition from school to work, could identify strategies for improving the  self-sufficiency of these beneficiaries and thus reduce their dependence on  the disability programs. Most of SSA’s current demonstration projects are  expected to continue until 2010 or later before generating final evaluation  results that could inform changes to disability program policy.
section_title: SSA Has Taken Steps to Improve Its Projects but Continues to Lack Management Controls
paragraphs: SSA has taken steps to improve its demonstration projects, in part by  applying more rigorous methodologies than it did for the projects SSA  initiated prior to 1998; however, it has not fully implemented GAO’s  recommendations from 2004 and does not have written policies and  procedures in place to ensure that projects are routinely reviewed and  effectively managed so that they yield reliable information about their  impacts. As a result, some projects faced challenges, such as low  participation rates or data collection problems, which were significant  enough to hinder the agency’s ability to evaluate the projects’ impacts as  planned. In addition, without comprehensive written policies and  procedures governing how SSA manages and operates its demonstration  program, the project objectives, designs, and evaluation plans may be  impacted during times of organizational change.
subsections: section_title: SSA Has Taken Steps to Improve Its Demonstration Projects
paragraphs: SSA has improved its demonstration projects by applying more rigorous  methodologies than it did prior to 1998, contracting with professional  researchers and appointing new management for the program.  Specifically, SSA is applying more rigorous evaluation methodologies to  the projects it has initiated since 1998 than it did to the projects initiated in  the late 1980s and early 1990s. At the time of our prior report, SSA officials  acknowledged that the limited rigor of those earlier projects reduced their  usefulness and indicated that the agency had placed a new emphasis on  ensuring that its projects going forward would be more rigorously  designed. Of the 14 projects that SSA has initiated since 1998, 13 were  early enough in the planning or design stages at that time to give SSA an  opportunity to make such improvements. Since that time, SSA has  completed much of the design work for its 14 projects and provided us  with detailed design information for 12 of them, enabling us to assess the  rigor of these projects’ designs for our current review.
Our current analysis shows that SSA did use more rigorous methodologies  for the projects initiated over the last decade than for its earlier projects.  SSA is now using methodologies known as experimental or quasi- experimental designs, which are commonly used by research professionals  conducting demonstration projects to estimate the impacts of program or  policy changes. On the basis of our assessment, we determined that 11 of  the 12 projects’ designs were strong or reasonable when assessed against  professional research standards (see table 3). We compared each  project’s design against GAO and recognized academic criteria for  conducting evaluation research, which were also consistent with statutory  requirements that DI projects be generally sufficient in scope and planned  in such a way to permit a thorough evaluation of the program or policy  changes under consideration. We also determined that the projects  currently under way could provide some reliable results if implemented  and evaluated as designed.
Despite this progress, we found that SSA did not always meet additional  DI and SSI statutory requirements regarding the general applicability of  the projects’ results and the use of expert advice, respectively. The  authorizing statute for DI demonstration projects requires that the results  derived from the projects will obtain generally in the operation of the  disability program. While one of the six DI projects, the BOND project,  has been designed to yield nationally representative information about the  impacts of the project, the statute does not require that the results be  applicable to all DI beneficiaries nationwide. However, the results should  apply to a larger group of beneficiaries than just those that participated in  the demonstration project, and SSA may be able to apply the results from  three other DI projects—the Accelerated Benefits demonstration project,  the Benefit Offset - 4 State Pilot, and the Mental Health Treatment Study— more generally because it plans to implement and evaluate the projects in  a consistent manner at multiple sites. In addition, one of the two jointly  authorized projects—the State Partnership Initiative—did not yield  generally applicable results because the projects were not implemented  consistently across each state. The authorizing statute for SSI projects  requires the Commissioner of SSA to obtain the advice and  recommendations of specialists who are competent to evaluate the  proposed projects as to the soundness of their design, the possibilities of  securing productive results, the adequacy of resources to conduct the  proposed research or demonstrations, and their relationship to other  similar research or demonstrations already completed or in process before  entering into a contract, grant, or cooperative agreement for the project.  However, SSA obtained advice from experts for only two of the six SSI  projects. Finally, SSA generally met other design criteria required by  statute for the BOND project (see app. II).
To further improve the demonstration projects’ planning and  methodological rigor, SSA has used external research professionals to  work with the agency on the design, implementation, or evaluation of 12 of  the 14 projects. SSA officials have acknowledged the need for additional  expertise to design and implement methodologically rigorous  demonstration projects. Thus, SSA has awarded, or planned to award,  contracts and cooperative agreements to research consultants and  universities with such expertise to evaluate 12 of its 14 projects (see app.  III). In nine cases, these researchers also worked on the design or  implementation of the projects. For example, for the Accelerated Benefits  demonstration project, an SSA research contractor also designed how and  where the project would be conducted and managed its implementation so  that the data needed for the evaluation it plans to conduct will be  available. We also found that SSA and most of these researchers  communicated regularly when collaborating on these projects and  researchers submitted monthly or quarterly progress reports to SSA,  which included information on expenditures, progress, and areas of  concern that needed to be addressed.
SSA also appointed new program management in 2007. Since that time, the  new management team has conducted an internal review of the 10  demonstration projects that were under way at the time of their  appointment. SSA officials told us that all projects underwent a thorough  review that was conducted by the Acting Associate Commissioner for  Program Development and Research and others with appropriate  expertise. Documents we obtained indicate that the review identified the  projects’ strengths, weaknesses, and whether they were likely to yield  reliable, useful results. For example, SSA considered whether a project’s  sample size and site selection were appropriate, if it had been  implemented in accordance with its design, and if it faced any challenges  that would prevent researchers from conducting a rigorous evaluation of  its results. SSA concluded that five of its projects would continue,  expecting that they were likely to yield reliable impact information; these  five projects are currently in progress. However, the agency did need to  make significant changes to strengthen one of the five projects’ designs— the BOND project. At an earlier point in the design phase, SSA expanded  the BOND project’s scope to include multiple components in addition to  the benefit offset, such as a health benefits package. The new management  team subsequently determined that the cost estimates and program  complexity associated with several of those components raised questions  about the feasibility of implementing the project and significantly scaled  back the scope of the study.
For the other five projects the new management was reviewing, SSA  determined they faced significant limitations or challenges—such as  poorly chosen implementation sites and low participation—that made it  highly unlikely for them to obtain reliable results or would have been  duplicative of other ongoing research (see table 4). Thus, SSA cancelled  those projects in 2007 and 2008. Although the only information SSA has  obtained from these projects are some lessons learned from the Pediatric  Medical Unit project, the agency projected that it would have spent  another $82 million had it not canceled them. Based on the information  SSA provided us about the challenges facing the projects and the expected  future costs of conducting them, its decisions to cancel the five projects  appear to have been data-driven and reasonable.
Further, SSA consolidated its research expertise by merging the Office of  Disability and Income Security Programs (DISP) with SSA’s Office of  Policy in February 2008, creating the Office of Retirement and Disability  Policy. As of June 2008, each office’s research unit remains intact and no  formal organizational changes were made to the demonstration program,  but agency officials told us that the merger has facilitated communication  and strengthened relationships between researchers within the agency.  For example, experts from the former Office of Policy’s research unit  routinely review and provide input on the demonstration projects’ designs  and evaluations.
section_title: SSA Continues to Lack Policies, Procedures, and Other Mechanisms to Ensure Projects Yield Reliable Information
paragraphs: While SSA is taking steps to generally improve its demonstration projects’  designs and address specific project limitations, it does not have policies,  procedures, and mechanisms to ensure that demonstration projects will  yield reliable information about the impacts of the programs they are  testing. According to internal control standards in the federal government,  federal agencies should have policies, procedures, and mechanisms in  place to provide reasonable assurance that a program’s objectives are  being achieved. However, we found that, as of August 2008, SSA had not  fully implemented the recommendations we made in 2004 to help ensure  the effectiveness of the demonstration projects. Specifically, SSA  continues to lack    a formal, comprehensive, long-term agenda for conducting demonstration    an expert panel to review and provide regular input on the design and  implementation of demonstration projects from the early stages of a  project through its final evaluation; and   a formal process for fully considering the potential policy implications of  its demonstration projects’ results and fully apprising Congress of the  results and their policy implications.
We did find that SSA has developed a limited research agenda for its  projects that lacks basic details about the projects, including their  objectives, schedules, and costs. The agenda was also developed without  broadly consulting key internal and external stakeholders to obtain their  input. SSA officials told us that they do not plan to update the agenda to  reflect that some of the projects have been cancelled. In contrast, other  federal agencies that conduct research have published much more detailed  research agendas and update them regularly. For example, the Department  of Education’s National Institute on Disability and Rehabilitation Research  publishes a 5-year plan that outlines priorities for rehabilitation research,  demonstration projects, training and related activities, explains the basis  for such activities, and publishes the plan for public comment before  submitting it to Congress.
In addition, agency officials said that SSA planned to continue using  experts only on an ad hoc basis, citing how potential conflicts of interest  could pose challenges to serve on an expert panel as we had  recommended. SSA established ad hoc panels or consulted with outside  experts for 8 of its 14 demonstration projects, including 4 of those that are  currently in progress. Many of SSA’s project officers and contractors  reported that this was a positive experience, and SSA management has  told us that they plan to continue using experts in the future. However,  under this approach SSA may miss the opportunity to obtain advice more  broadly on the demonstration program. For example, the panels and  experts that SSA used were brought on board after the agency decided to  initiate the demonstration projects; therefore, they were not in position at  an early enough point in time to help SSA consider whether a  demonstration project or an alternative research approach was, in fact, the  best way to meet the agency’s needs. Furthermore, SSA did not regularly  seek input from the Social Security Advisory Board or the National  Council on Disability, which both play key roles in federal disability policy  and could be in a position to advise SSA more broadly on the  demonstration projects.
In addition, our prior work found that SSA had not sufficiently provided  information on the status and results of its demonstration projects to  Congress. In our current review, we found that SSA regularly submits  annual reports about the DI demonstration projects to its congressional  oversight committees. While SSA meets its statutory requirements by  submitting these reports, the information in them is generally limited to  descriptions of the projects’ objectives and the dates of upcoming  milestones. Similarly, the information that SSA reports about its SSI  demonstration projects is limited to brief descriptions in the agency’s  annual congressional budget justifications. Key information that could  help Congress monitor the progress of the demonstration projects— including project costs, potential risks and obstacles to their success, or  the policy implications of their results—was rarely included in the annual  reports or budget justifications. However, SSA officials also told us that  they sometimes share additional information with Congress about the  demonstration projects. For example, SSA officials told us they met with  congressional committees in October and November 2007 to share  information about its design plans for the BOND project.
In addition to not fully addressing our prior recommendations, SSA does  not have written policies and procedures governing how it should review  and operate its demonstration program. Accordingly, SSA does not have a  written policy requiring SSA management to review its project officer’s  demonstration projects on a regular basis. Standards for internal control in  the federal government state that managers should compare a program’s  actual performance against expected targets and analyze significant  differences. Although the new program management team reviewed each  of the demonstration projects at the time of their appointment, SSA does  not have a written policy requiring such a review process periodically  throughout the design, implementation, and evaluation phases of each  project. SSA’s lack of a policy to systematically review each project on a  periodic basis contributed to problems sometimes going undetected after  the projects were implemented, and they did not yield the data needed for  their evaluations. For example, because SSA was not actively involved in  implementing or monitoring the Florida Freedom Initiative demonstration  project, it was not in a position to take steps to ensure that the project  proceeded as planned, and staff at the state level failed to enroll enough  participants to generate data for the evaluation that SSA planned.  Therefore, no evaluation was conducted for this project.
In addition, we found that SSA does not have written procedures for its  project officers to follow as they design, implement, and evaluate its  demonstration projects. Such procedures could be used to ensure that  standard research practices, such as conducting pilot phases and including  internal and external stakeholders, are applied when planning and  implementing the demonstration projects. Specifically, SSA does not  require staff to regularly use pilots to test projects’ underlying  assumptions, operational logistics, or feasibility before they are  implemented. As a result, SSA planned or conducted pilots or phased  implementations for only 8 of its 14 projects, although GAO criteria for  evaluation research emphasize the importance of conducting pilots, as  they are a critical test of a project’s design. At least four of the projects  that did not include pilot phases experienced the type of logistical  challenges that pilots are intended to identify. For example, the Homeless  Outreach Projects and Evaluation demonstration project experienced  start-up delays because of compatibility problems between the  contractor’s online data collection system and the computer systems at the  41 sites where the project was implemented. If SSA had conducted a pilot  phase for this project, it may have detected these issues at a smaller  number of sites and developed a plan to resolve them prior to  implementing the full project.
In addition, SSA does not have written procedures directing its project  officers or contractors to routinely consult with internal and external  stakeholders when planning the demonstration projects. We found that at  least 11 of the 14 projects experienced challenges or limitations because  SSA had not obtained sufficient input from, or coordinated effectively  with, internal and external stakeholders. For example, SSA officials told us  that they were aware that the Benefit Offset - 4 State Pilot project and the  Benefit Offset National Demonstration project required a change to  internal SSA processes for calculating DI benefits but did not coordinate  with key internal stakeholders early on to determine how to make this  change, and no systematic process was put in place. As a result, SSA had  to calculate payments by hand for the Benefit Offset - 4 State Pilot, and the  BOND project’s implementation has been delayed while SSA now works  with its internal stakeholders to determine how to make the needed  changes. Furthermore, SSA officials, contractors, and representatives from  the various demonstration projects’ implementation sites told us that there  was little input from internal stakeholders and that internal coordination  problems existed on at least seven of these projects, including three of  those that were cancelled. We also found that lack of coordination or  communication with external stakeholders led to challenges in at least  seven of these projects, including four of those that were cancelled. For  example, coordination problems between the two contractors for the  California Rise project resulted in the project’s components being  designed in isolation from each other, which complicated the evaluation  plans and eventually contributed to the project’s cancellation. In addition,  SSA did not always include the prospective implementation sites in the  planning and design of its projects, although they could have provided  insight into the feasibility and logistical requirements of the project.
While SSA has periodically provided direction informally to its project  officers, some project officers told us that more formal guidance would  have helped them to better understand what steps were necessary and  expected, and we concluded from our discussions with others that such  guidance would have been helpful. SSA recognizes that the program’s lack  of written procedures is a limitation and is drafting a guidebook on  standard research practices for staff to follow when planning and  designing demonstration projects. Although an SSA official told us that  this document is a work in progress, it appears that the handbook will  include key procedures for designing a project, such as identifying what  data is needed for the evaluation and how it should be obtained. It also  directs project officers to assemble a team for project development that  includes staff from across the agency who will be able to collaborate and  provide the input necessary to address the multiple components of the  demonstration project. The draft guidebook also includes provisions for  assembling a research panel composed of internal and external experts.  This panel would review proposed research projects and identify those  that present the most promising opportunities, taking into consideration  the extent to which prior research has already addressed the topic.  However, in its current form, the guidebook provides little direction for  the implementation or evaluation phases of the demonstration projects,  and SSA officials had not finalized it as of May 2008.
Without comprehensive written policies and procedures governing how  SSA manages and operates its demonstration programs, the project  objectives, designs, and evaluation plans may be impacted during times of  organizational change. Because government operating conditions  continually change, agencies should have mechanisms in place to identify  and address any special risks arising from such changes, especially those  caused by hiring new personnel to occupy key positions in the agency.  However, because SSA lacks mechanisms such as a standing advisory  panel or written policies and procedures to provide continuity for its  demonstration program when organizational changes occur, it cannot  guarantee that institutional knowledge about the projects is shared or that  the impacts of such changes are considered as the projects progress. SSA  has experienced several organizational changes since the first of these  projects was initiated in 1998, which have included the demonstration  program’s relocation from the Office of Policy to the Office of Disability  and Income Support Programs in 2002, program management’s  replacement in 2007, and the Office of Policy and Office of Disability and  Income Support Program’s merger in 2008. At least six of SSA’s projects  experienced schedule delays and cancellations, in part because newly  appointed officials made significant changes to some projects or  determined that because others faced significant limitations or potential  challenges it was not in the agency’s interests to continue them (see fig. 1).  While certain management actions may be reasonable, SSA’s lack of  written policies and procedures governing how such steps are taken  leaves current and future projects vulnerable to disruption. For example,  we found that the Benefit Offset National Demonstration project is still in  the design phase after 9 years, during which time it has gone through  numerous revisions by different program managers and was moved from  one office to another. As of August 2008, an interagency working group  was determining how to implement this administratively complex project.  SSA has put the project’s implementation and evaluation on hold until this  issue has been resolved.
section_title: Conclusions
paragraphs: For over two decades, SSA has had the authority to conduct  demonstration projects to test strategies that could address the challenges  posed by the low rate of return into the workforce and the growing  number of applicants and disabled beneficiaries. However, the agency has  missed opportunities to identify ways to modernize DI and SSI programs  and policies because it has generally not conducted the demonstration  projects effectively. Since 1998 alone, SSA has spent over $150 million on  14 demonstration projects; yet these projects have generated limited  information about the impacts of the strategies that were being tested.  Although many of these projects were generally well designed, SSA’s lack  of written polices, procedures, and mechanisms for managing and  operating these projects is one of the key reasons the projects SSA has  completed and cancelled to date were generally not implemented and  evaluated in a way that yielded reliable, data-driven impact information.  As a result, Congress, SSA, and other organizations that play a critical role  in federal disability policy continue to lack key information about  important issues, such as the impact of providing health care or  employment supports to DI and SSI beneficiaries as a means to help  beneficiaries achieve self sufficiency and leave the rolls.
SSA’s five demonstration projects currently in progress have the potential  to identify solutions to some of SSA’s challenges. However, if SSA does  not address the limitations in the way it manages and operates  demonstration projects, these projects may encounter the same challenges  that past projects have faced, and SSA could again have little to show for  its efforts. Given that SSA estimates it will spend approximately $220  million over the next several years to complete these projects, it is  important that steps be taken to make the projects less vulnerable to the  challenges and organizational changes they could encounter in the future.  SSA’s actions to review its demonstration projects and to begin drafting  guidance to help staff better plan and design its projects are encouraging  first steps. As SSA officials work toward finalizing this guidance, it is also  necessary for the agency to address its lack of written policies and  procedures for managing and operating its projects during their  implementation and evaluation phases. SSA should also take action to  fully implement the recommendations we made in 2004. Implementing  those recommendations by fully developing its research agenda,  establishing an expert panel to advise it about the projects on a regular  basis, and improving its communications with Congress could help  improve the effectiveness and transparency of its demonstration program  going forward.
section_title: Recommendation for Executive Action
paragraphs: To improve SSA’s management of its demonstration projects we  recommend that the Commissioner of Social Security direct the Deputy  Commissioner for the Office of Retirement and Disability Policy to    establish written policies, procedures, and mechanisms for managing and  operating its demonstration projects that are consistent with standard  research practices and internal control standards in the federal  government, including those for coordinating with internal and external  stakeholders and sharing information with Congress.
section_title: Agency Comments and Our Evaluation
paragraphs: We obtained written comments on a draft of this report from SSA, which  are reproduced in appendix IV. We incorporated technical comments we  received throughout the report, as appropriate.
In response to our draft report, SSA generally agreed with our  recommendation and acknowledged the need to develop a guidebook to  assist its staff in the design, implementation, and evaluation phases of its  demonstration projects. SSA further discussed its existing processes and  written procedures for managing and reviewing its programs, including the  demonstration project program. While noteworthy, we continue to believe  SSA needs to establish written procedures that incorporate professional  research standards and internal control mechanisms for ensuring that the  demonstration projects yield reliable information about their impacts. SSA  considers its current guidebook a work in progress. Further, SSA stated  that the agency has taken steps in recent years to address our prior  recommendations. While we acknowledge SSA’s efforts, the agency needs  to take additional steps to fully implement them. For example, SSA  continues to lack a standing expert panel to review and provide regular  input on the demonstration projects overall, even though it has employed  subject matter experts for some of its demonstration projects. Although  SSA officials have raised concerns about the difficulty of establishing an  expert panel because research contractors serving on the panel would be  precluded from working on individual projects, we continue to believe that  such a panel could be established. As previously recommended, this panel  should also include SSA’s key research personnel and outside disability  experts in addition to researchers.
We are sending copies of this report to the Commissioner of SSA and other  interested parties. We will also make copies available to others upon  request. In addition, the report will be available at no charge on the GAO  Web site at http://www.gao.gov.
If you or your staffs have any questions concerning this report, please  contact me at (202) 512-7215. Contact points for our offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine how SSA has used its current demonstration authority, we  reviewed legislation authorizing the Social Security Administration (SSA)  to conduct Disability Insurance (DI) and Supplemental Security Income  (SSI) demonstration projects, prior GAO report and SSA’s Office of the  Inspector General (OIG) reports, and reports to Congress on the  demonstration authority. We reviewed documents from SSA and from  publicly available sources, including the Federal Register and reports by  other research organizations. We interviewed current and former SSA  officials in the Office of Disability and Income Security Programs,  specifically, the Office of Program Development and Research (OPDR)  and the Office of Research, Evaluation and Statistics who had  responsibility for, or involvement in, the demonstration projects. We also  interviewed research contractors that worked on the demonstration  projects and individuals from organizations that have a key role in federal  disability policy. In addition, we interviewed staff from sites where SSA  implemented 9 of the 14 demonstration projects. We selected sites that  included ongoing, cancelled and completed projects, and represented  diverse geographic regions throughout the United States.
To better understand SSA’s DI and SSI demonstration projects, we  reviewed SSA documents describing the purpose, design, and status for all  demonstration projects that were in progress, completed, or had been  cancelled prior to completion. These documents included requests for  proposals, project plans and schedules, interim or final project reports,  and reports to Congress from 1998 to 2008. We used these documents to  identify key characteristics of the projects, including the policy issues  addressed, use of contractors, the authority used to conduct each project,  project timelines, and information resulting from each project. We also  examined the issues SSA tested or was in the process of testing in its  demonstration program. We reviewed the authorizing statutes for the DI  and SSI demonstration programs, as well as requirements for specific  demonstrations included in the Ticket to Work and Work Incentives  Improvement Act of 1999 to determine the extent to which the projects in  SSA’s demonstration program address statutory requirements. For  projects cancelled during this time period, we collected cancellation  memos and other documentation to determine SSA’s reasons for the  cancellations.
To describe the costs associated with the program, we collected  expenditures data from SSA for each project—including funds spent to  date for each project and total anticipated funding for the projects that are  currently in progress. We assessed the reliability of the budget data by (1)  manually checking the required data elements, (2) reviewing existing  information about the data and the system that produced them, and (3)  interviewing agency officials knowledgeable about the data. When we  found discrepancies, we brought them to the attention of SSA officials and  worked with them to correct the discrepancies before conducting our  analyses. Based on these efforts, we determined that the data were  sufficiently reliable for the purposes of this report.
To assess the extent to which SSA’s demonstration projects were designed  in accordance with professional research standards and statutory criteria,  we reviewed the most current information that SSA provided about each  project, either an evaluation design or final evaluation report. These  design and evaluation methodologies were assessed against professional  research standards, consistent with the authorizing statutes’  methodological requirements and GAO’s and recognized academic criteria  for conducting evaluation research. Key components of the professional  research standards include    methodological rigor of the project’s design and evaluation and their  appropriateness given the purpose of the research (e.g. use of an  experimental or quasi-experimental design for an impact evaluation);   appropriate handling of any problems encountered when implementing  the evaluation’s design, such as participant attrition or insufficient sample  sizes;   appropriate handling of any problems encountered with the data, such as  missing values or variables;    appropriate variables to ensure internal and external validity, given the  evaluation’s design;   appropriate data analysis and statistical models, such as frequencies or  multivariate analysis, given the evaluation’s design; and   overall strength of the evaluation design and analysis.
To assess the appropriateness of each study’s methodology for answering  the research questions, we developed two data collection instruments  based on these professional research standards—one for evaluation  designs and one for the final evaluation reports. We then examined the  strengths and weaknesses of the evaluation designs and final reports,  taking into consideration the project’s objectives, resource constraints,  methodological approach, technical adequacy of plans for data collection  and analysis, and when available, the presentation of the findings. A social  scientist read and coded each evaluation design or final report. A second  social scientist reviewed each completed data collection instrument and  the relevant documentation to verify the accuracy of every coded item. For  each DI demonstration project, we also reviewed the reports to ascertain  whether they met statutory requirements that the project’s results be  broadly applicable to relevant segments of the DI beneficiary population,  not just the project participants. For each SSI demonstration project, we  also interviewed agency officials to determine whether SSA met its  statutory obligation to obtain the advice and recommendations of  specialists who are competent to evaluate the projects as to the soundness  of their design, the possibilities of securing productive results, the  adequacy of resources to conduct them, and their relationship to other  similar research or demonstrations already in progress. We also identified  key provisions of the demonstration authority statutes to assess SSA’s  compliance with congressional reporting requirements.
To address SSA’s planning and management of its demonstration projects,  we interviewed SSA management and staff about the agency’s policies,  guidance and processes on developing and implementing demonstration  projects, and collected supporting documentation where available. We  assessed the adequacy of SSA’s internal controls using the criteria in  GAO’s Standards for Internal Control in the Federal Government,  GAO/AIMD 00-21.3.1, dated November 1999. These standards, issued  pursuant to the requirements of the Federal Managers’ Financial Integrity  Act of 1982 (FMFIA), provide the overall framework for establishing and  maintaining internal control in the federal government. Also pursuant to  FMFIA, the Office of Management and Budget issued Circular A-123,  revised December 21, 2004, to provide the specific requirements for  assessing the reporting on internal controls. Internal control standards and  the definition of internal control in Circular A-123 are based on the GAO  Standards for Internal Control in the Federal Government.
section_title: Appendix II: Additional Statutory Criteria for the Benefit Offset National Demonstration Project
paragraphs: In addition to meeting professional research standards and the mandated  methodological requirements for the DI demonstration projects, the  Benefit Offset National Demonstration (BOND) project is required to  address a number of specific issues. We found that BOND met all of these  requirements except the one to test the project’s effect on induced entry to  and reduced exit from the DI rolls (table 5).
In addition, while not included as part of the BOND demonstration, SSA  plans to conduct research on variations in the amount of the offset as a  proportion of earned income to determine the appropriate offset  disregard, in accordance with the DI authorizing statute’s requirements.
section_title: Appendix III: SSA’s Use of Research Contractors and Universities on the Demonstration Projects
subsections: section_title: Research consultant or university
paragraphs: Projects conducted under DI demonstration authority  (prime contractor)
Mathematica Policy  Research  (subcontractor)
SSA awarded MDRC a contract in 2006 for the project’s design, implementation,  and evaluation. MDRC’s design for the project and evaluation expanded on  preliminary design criteria SSA developed between 2004 and 2005, and published  in a Request for Proposal. MDRC finalized the design in collaboration with  Mathematica Policy Research and SSA. MDRC and Mathematica began enrolling  participants in the project in October 2007. Implementation will continue through  2011. MDRC is conducting the impact evaluation and Mathematica is conducting the  process evaluation. SSA expects the evaluation to be completed in 2011.
SSA awarded Abt Associates a contract in 2004 for the project’s design and will  award Abt a second contract for its implementation and evaluation contingent on  successful completion of the design. Abt’s design for the project and evaluation is  based on preliminary design criteria SSA developed between 1999 and 2004 and  published in a Request for Proposal. Abt is finalizing the design and planning for  implementation in collaboration with SSA.
SSA awarded Mathematica Policy Research a contract in 2006 to design and  implement the health services part of the project and to design and conduct the  evaluation. Mathematica’s design for the heath services component and the  evaluation is based on preliminary design criteria SSA developed between 2004 and  2006 and published in a Request for Proposal. Mathematica worked on the design  in 2006 and 2007, and SSA reviewed and provided input on it. Mathematica  expected to implement the project in 2007 and complete its evaluation in 2011, but  SSA cancelled the project in 2007 before implementation began.
Disability Research  Institute (Rutgers  University)
SSA awarded the Disability Research Institute a contract in 2000 for Rutgers  University to design, implement, and evaluate an Early Intervention pilot project.  Rutgers proposed a design for the project and evaluation and submitted an  evaluation design report for SSA’s approval in 2002. Rutgers planned to implement  the project in 2003 and begin its evaluation in 2004, but the project remained under  design until the contract expired in 2005.
SSA modified Abt’s contract for the BOND project in 2005 to incorporate Early  Intervention into BOND’s design, implementation, and evaluation. Abt proposed  options to incorporate the Early Intervention project, but SSA later decided to cancel  it and eliminated it from Abt’s contract in 2007.
SSA awarded a contract to Westat in 2005 to implement and evaluate a project SSA  designed between 2003 and 2005. Westat began working with the project sites in  2005 to prepare for implementation and began enrolling participants in the project in  2006. Implementation is scheduled to continue through 2010. Westat began the  evaluation in 2008 and will complete it in 2011.
DOL awarded a contract to the University of Iowa College of Law, Law Health Policy  and Disability Center to provide technical assistance for the project’s implementation  and to evaluate the project. SSA provided partial funding for the contract through an  Interagency Agreement with DOL in 2002, 2003, and 2004. Implementation began in  2003. The University of Iowa did not evaluate the impact of the project because it  could not meet SSA’s data security requirements in order to obtain data needed for  the evaluation. SSA officials told us that DOL plans to evaluate the project under a  new contract with Mathematica Policy Research, but SSA is not funding that  evaluation.
SSA awarded a contract to George Washington University in 2004 to conduct  preliminary research related to the Early Identification and Intervention project. SSA  designed the demonstration project between 2004 and 2006 and issued a Request  for Proposal for its implementation in 2007, but did not award a contract to  implement or evaluate the project.
SSA funded a contract for Mathematica to design an evaluation for the project  through an Interagency Agreement with the Department of Health and Human  Services in 2004. Due to low enrollment, data needed for the evaluation was not  collected by the State of Florida, which designed and implemented the project  between 2003 and 2007, and the evaluation was not conducted.
SSA awarded a contract to Westat in 2004 to evaluate a project SSA designed  between 2003 and 2004. SSA implemented the project in 2004. Westat completed  its evaluation in 2007.
SSA planned to issue a Request for Proposal to hire a professional research  organization to design, implement, and evaluate the project but cancelled it before it  awarded the contract.
University Centers on  Disabilities (AUCD)
SSA awarded a contract to AUCD in 2006 to design, implement, and evaluate the  project; however, AUCD’s design for the project began in 2004 under an extension  of another contract. AUCD partially implemented the project in 2006 and 2007. SSA  cancelled the project in 2008 before AUCD completed the implementation or  evaluation.
Projects jointly authorized under DI and SSI authorities  Virginia Commonwealth  University (VCU)  (prime contractor)
Mathematica Policy  Research  (subcontractor)
SSA awarded multiple contracts to VCU starting in 1998 to provide technical  assistance to the states that were implementing the project and to conduct an  evaluation. VCU completed its evaluation in 2006. SSA awarded three contracts  executed by VCU’s subcontractor, Mathematica: one in 1998 to provide technical  assistance to the states that were implementing the project, another in 1999 to  design the evaluation, and one in 2003 to conduct the evaluation. Mathematica  completed its evaluation in 2005.
Mathematica Policy  Research  (prime contractor)
MDRC (subcontractor)
SSA originally designed and implemented the project in 2003, but due to  methodological limitations, SSA awarded a contract to Mathematica in 2005 to  redesign, implement, and evaluate the project. Mathematica began implementing  the new design in 2006. Implementation will continue until 2013. Mathematica will  complete its evaluation in 2014.
section_title: Appendix IV: Comments from the Social Security Administration
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, key contributions to this report  were made by Michael Collins, Assistant Director; Jason Holsclaw and  Anne Welch, Analysts-in-Charge; Dana Hopings; Annamarie Lopata; Jean  McSween. Additional support was provided by Kenneth Bombara; Daniel  Concepcion; Jennifer Cook; Cindy Gilbert; Sharon Hermes; Joanie Lofgren;  Joel Marus; Mimi Nguyen; Patricia Owens; Daniel Schwimer; Kris  Trueblood; Kathy White; Charles Willson; Elizabeth Wood; and Jill Yost.
section_title: Related GAO Products
paragraphs: Federal Disability Programs: More Strategic Coordination Could Help  Overcome Challenges to Needed Transformation. GAO-08-635.  Washington, D.C.: May 20, 2008.
Social Security Disability: Better Planning, Management, and  Evaluation Could Help Address Backlogs. GAO-08-40. Washington, D.C.:  December 7, 2007.
High Risk Series: An Update. GAO-07-310. Washington, D.C.: January  2007.
Federal Disability Assistance: Wide Array of Programs Needs to be  Examined in Light of 21st Century Challenges. GAO-05-626. Washington,  D.C.: June 2, 2005.
Social Security Disability: Improved Processes for Planning and  Conducting Demonstrations May Help SSA More Effectively Use Its  Demonstration Authority. GAO-05-19. Washington, D.C.: November 4,  2004.
SSA’s Rehabilitation Programs. GAO/HEHS-95-253R. Washington, D.C.:  September 7, 1995.
Impact of Vocational Rehabilitation Services on the Social Security  Disability Insurance (DI) Program. GAO/HRD-T-88-16. Washington, D.C.:  May 26, 1988.
Social Security: Little Success Achieved in Rehabilitating Disabled  Beneficiaries. GAO/HRD-88-11. Washington, D.C.: December 7, 1987.
Social Security: Observations on Demonstration Interviews with  Disability Claimants. GAO/HRD-88-22BR. Washington, D.C.: December 3,  1987.
Social Security: Demonstration Projects Concerning Interviews with  Disability Claimants. GAO/HRD-87-35. Washington, D.C.: February 19,  1987.