Social Security Disability: Management Controls Needed to Strengthen Demonstration Projects
Released: 2008-09-26T00:00:00

Letter

Background

Under the Social Security Act of 1935, as amended, SSA administers two federal disability programs—DI and SSI—intended to provide benefits to individuals with disabilities who are unable to work. The DI insurance program provides monthly cash benefits to individuals who have a Social Security work record and the amount of benefits is related to prior earnings. The DI program is funded primarily through a payroll tax required by the Federal Insurance Contributions Act (FICA) and is levied on most workers, and DI benefits are based on an individual’s career earnings. The SSI program is a means-tested entitlement program that provides monthly benefits to aged, blind, or disabled individuals who have very limited income and assets. The SSI program is funded through general revenues. Unlike the DI benefit, the federal SSI benefit is a flat amount (adjusted for other income the individual may have) and is not related to prior earnings.

During the 1970s, as the number of disability awards and costs were increasing significantly for the DI program, Congress enacted legislation providing various work incentives to encourage beneficiaries to return to work and, potentially, leave the benefit rolls. To further these efforts, in 1980, Congress provided SSA with the authority to conduct demonstration projects to evaluate the effectiveness of policy alternatives that could encourage both DI and SSI beneficiaries to re-enter the workforce. Under this authority, SSA can temporarily waive DI and SSI program rules, including rules regarding program eligibility and benefit administration, in order to test the effect certain program changes would have on beneficiaries’ return-to-work rates and the size of the DI and SSI benefit rolls. Because Congress has historically granted SSA DI demonstration authority on a temporary basis, it is subject to periodic review and renewal. Since first providing this authority in 1980, Congress has renewed it in 1986, 1989, 1994, 1999, and 2004. However, in 2004, Congress only extended SSA’s DI demonstration authority through December 2005. As a result, SSA cannot initiate new DI demonstration projects but can continue those projects that were initiated on or before the December 2005 expiration date. However, SSA can continue to initiate demonstration projects under its SSI authority. In 2008, SSA requested that Congress reauthorize its DI demonstration authority, and a bill was introduced to do so.

SSA’s DI demonstration projects—unlike other SSA research activities— are paid for via the DI trust fund. Therefore, SSA is not required to obtain congressional approval for DI demonstration expenditures, although it is required to obtain approval from the Office of Management and Budget for an annual apportionment of the trust funds for these demonstrations. Unlike the DI projects, SSI demonstration projects are funded from SSA’s overall congressional research appropriation. Although SSA’s DI and SSI demonstration authorities are separate, the agency’s disability demonstration projects are sometimes jointly authorized when they involve both DI and SSI beneficiaries and applicants. When a demonstration project is conducted jointly under the DI and SSI demonstration authorities, funding for the project is split between trust fund (i.e., DI) and appropriated (i.e., SSI) sources.

SSA’s Office of Program Development and Research (OPDR) provides program analysis in support of DI and SSI programs. As part of their responsibilities, OPDR—sometimes with the assistance of outside research organizations—identifies the requirements for individual disability program demonstration projects, including the basic objectives, scope, and methodological standards for these projects. OPDR project officers are primarily responsible for overseeing the projects to ensure that they meet SSA’s technical and programmatic requirements.

As we have previously reported, demonstration projects examining the impact of social programs aim to provide evidence of the feasibility or effectiveness of a new approach or practice and are inherently complex and difficult to conduct. Measuring outcomes, ensuring the consistency and quality of data collected at various site locations, establishing a causal connection between outcomes and program activities, and separating out the influence of extraneous factors can raise formidable technical and logistical problems. Although the legislation granting SSA its demonstration authority does not require the use of particular methodological approaches, SSA has historically recognized that the law’s general requirement for its demonstration projects requires SSA to conduct its projects in a rigorous manner that provides the agency with a reliable basis for making policy recommendations. According to professional research standards, a rigorous study should include a clearly stated research question and methodology, including plans for data collection and evaluation, as well as appropriate controls to determine if a relationship exists between observed outcomes and the program change under examination (see app. I).

As part of our prior work related to SSA DI demonstration authority we reviewed two DI demonstration projects that SSA conducted in the late 1980s and early 1990s. At that time, we found that SSA had not used its demonstration authority to extensively evaluate a wide range of DI policy areas dealing with return to work and found that the demonstration projects had little impact on SSA’s and Congress’s consideration of DI policy issues. To facilitate close congressional oversight and provide greater assurance that SSA will make effective use of its authority, we recommended that SSA develop a formal agenda for its demonstration projects, establish an expert panel to guide the design and implementation of its demonstration projects, and establish formal processes to ensure full consideration of demonstration project results. We also identified several matters for Congress to consider, including continuation of DI demonstration authority on a temporary basis, establishment of additional reporting requirements for demonstrations, and clearer specification of the methodological and evaluation requirements of demonstrations.

SSA Has Used Its Demonstration Authority to Initiate Several Projects Since 1998, but Efforts Have Yielded Limited Information

Over the last decade, SSA has initiated 14 demonstration projects to test policy and program changes, of which SSA has completed 4, cancelled 5, and had 5 projects in progress as of August 2008. In total, SSA had spent about $155 million on these demonstration projects as of April 2008, and officials anticipate spending another $220 million in the coming years on those projects currently in progress. However, these projects have yielded limited information for influencing program and policy decisions. We found that SSA did not conduct impact evaluations for two of its completed projects and cancelled five projects prior to conducting formal evaluations; thus, limited information is available.

Since 1998, SSA Has Initiated 14 DI and SSI Demonstration Projects in an Effort to Test Policy and Administrative Changes

Since 1998, SSA has initiated 14 projects under its demonstration authority to test both DI and SSI program changes—6 related to DI, 6 related to SSI, and 2 examining both programs jointly (see table 1 for an overview of each project). As of April 2008, SSA spent $80.3 million on its completed projects, $7.1 million on cancelled projects, and $68.2 million on those currently in progress.

While SSA initiated 14 projects over the past 10 years, the agency has only completed 4 of them to date. These completed projects generally focused on reducing individuals’ dependency on the SSI program by primarily testing program waivers and other changes in program administration, as outlined in its SSI demonstration authority. We also found that SSA cancelled five projects during this period, citing significant challenges that would have limited the agency’s ability to obtain reliable information from them.

SSA had five projects in progress as of August 2008. These projects generally addressed topics outlined in the authorizing legislation for DI demonstrations and included strategies to return individuals to work and reduce the growth of certain subgroups of beneficiaries. For example, the legislation required projects to test various incentives to increase DI beneficiaries work activity. In addition, the Ticket to Work and Work Incentives Improvement Act of 1999 provided demonstration authority for a benefit reduction, rather than complete benefit termination, when beneficiaries had earnings that exceeded a certain level. To address this provision, SSA initiated the Benefit Offset National Demonstration (BOND) shortly after passage of the statute. Another project in progress, the Mental Health Treatment Study, is focused on identifying strategies for providing mental health treatment and employment supports for certain DI beneficiaries with mental illnesses. As of April 2008, officials estimated that the total costs for the five projects currently in progress would be about $288 million—about $220 million more than the $68 million already expended (see table 2).

Demonstration Projects Have Yielded Limited Information to Influence Program and Policy Changes

Despite using its demonstration authority to examine various issues, SSA’s demonstration projects have yielded limited information for influencing program and policy decisions. As required under its demonstration authority, SSA’s demonstration projects should be conducted in such a way to permit a thorough evaluation of alternative methods under consideration. However, we found that SSA had not conducted impact evaluations—assessments of a project’s effects compared to what would have happened in its absence—for two of its completed projects, the Disability Program Navigator and the Florida Freedom Initiative. Thus, no information about the impacts of the program and policy changes being tested was available for making decisions about disability policy. The Disability Program Navigator project, which SSA conducted with the Department of Labor (DOL), was not evaluated because the evaluation contractor could not meet SSA’s data security requirements established after the project was already in progress, and thus could not access the necessary data. SSA developed a plan to evaluate the Florida Freedom Initiative after they became concerned about the state’s evaluation plans. However, SSA did not conduct an evaluation because staff at the state level conducting the project did not enroll enough participants in the project to meet sample size requirements. Thus, there was not enough data available to conduct a reliable evaluation.

Furthermore, SSA intended to evaluate the impacts of policies and programs being tested in five other projects but could not do so because the significant challenges those projects faced led SSA to cancel them in the early stages. Specifically, four of these projects were cancelled prior to implementation, and thus no data was available to conduct the evaluations of those policy and programs being tested. The other cancelled project— the Pediatric Medical Unit demonstration—was partially implemented but not evaluated because the project did not establish the comparison group needed for the analysis. The project also did not it enroll enough participants at some implementation sites to meet the sample size requirements needed to generate data for a reliable evaluation. However, SSA was able to obtain some preliminary information on how the project’s strategy appeared to be working at two site locations and is considering how to use it.

Although SSA did conduct evaluations for two of the completed projects— the Homeless Outreach Projects and Evaluation (HOPE) project and the State Partnership Initiative project—we found that these projects also yielded little information about the impacts of the strategies being tested because the reported evaluation results could not reliably demonstrate the projects’ effects. For example, an outcome evaluation of the HOPE project showed that although disability program applicants assisted by the project received faster decisions from SSA about whether to allow or deny benefits, another federal agency initiated a similar project even though the HOPE project was under way. Therefore, SSA’s evaluation results were weakened, in part because researchers could not separate the effects of the SSA project from the effects of the other federal project. While SSA did not obtain reliable impact evaluation results from this project, agency officials told us that they did obtain a great deal of information about the process of conducting this type of demonstration project.

For the State Partnership Initiative, we found that SSA did conduct an impact evaluation when the project ended, but data available at that time were incomplete, and thus information about the impact of the project may not be a reliable indicator of the project’s long-term effects. SSA’s contractors recommended that a final evaluation be conducted once all the data were collected to assess whether the preliminary results were valid. However, SSA management chose not to pursue further evaluation because the preliminary results indicated that the project was not successful at increasing earnings enough to allow individuals returning to work to exit the rolls and no longer be dependent on disability benefits. Nonetheless, SSA’s contractors and agency officials said that lessons learned from implementing the State Partnership Initiative have influenced the agency’s subsequent approach to return beneficiaries to work. For example, SSA used the job descriptions of benefits planners, as well as data systems from this project, to design the agency’s national Benefits Planning and Outreach program.

SSA has also begun to obtain some information from one of the five projects currently under way. SSA has used preliminary results of the Benefit Offset - 4 State Pilot to aid in the design of the BOND project. Each of the four states conducting this pilot has provided an interim report to SSA detailing lessons learned from the implementation of this project. Because the pilot and BOND both test a benefit offset in conjunction with other DI program changes, SSA officials and the BOND project contractor believe that states’ experiences implementing this pilot will help SSA identify and resolve operational issues before rolling BOND out nationally. In addition, the four states have conducted preliminary impact evaluations for the pilot project and expect to complete final evaluations once the project’s implementation and data collection phases are over.

SSA also plans to conduct impact evaluations of the other demonstration projects it had in progress as of August 2008. While they have the potential to yield reliable results, it is too early to tell whether they will ultimately be useful for informing DI and SSI policy and program changes. These projects address issues outlined in the demonstration authority statutes and disability programs more broadly, and SSA officials believe they will yield useful information. For example, SSA officials anticipate that the results of the Accelerated Benefits demonstration project could help policymakers determine whether to eliminate the 24 month waiting period for Medicare that DI beneficiaries encounter under current law. SSA officials also anticipate that demonstration projects in progress could yield key information on how to improve outcomes for certain subgroups of beneficiaries. For example, SSA officials said that the Youth Transition Demonstration, which targets young people with disabilities as they transition from school to work, could identify strategies for improving the self-sufficiency of these beneficiaries and thus reduce their dependence on the disability programs. Most of SSA’s current demonstration projects are expected to continue until 2010 or later before generating final evaluation results that could inform changes to disability program policy.

SSA Has Taken Steps to Improve Its Projects but Continues to Lack Management Controls

SSA has taken steps to improve its demonstration projects, in part by applying more rigorous methodologies than it did for the projects SSA initiated prior to 1998; however, it has not fully implemented GAO’s recommendations from 2004 and does not have written policies and procedures in place to ensure that projects are routinely reviewed and effectively managed so that they yield reliable information about their impacts. As a result, some projects faced challenges, such as low participation rates or data collection problems, which were significant enough to hinder the agency’s ability to evaluate the projects’ impacts as planned. In addition, without comprehensive written policies and procedures governing how SSA manages and operates its demonstration program, the project objectives, designs, and evaluation plans may be impacted during times of organizational change.

SSA Has Taken Steps to Improve Its Demonstration Projects

SSA has improved its demonstration projects by applying more rigorous methodologies than it did prior to 1998, contracting with professional researchers and appointing new management for the program. Specifically, SSA is applying more rigorous evaluation methodologies to the projects it has initiated since 1998 than it did to the projects initiated in the late 1980s and early 1990s. At the time of our prior report, SSA officials acknowledged that the limited rigor of those earlier projects reduced their usefulness and indicated that the agency had placed a new emphasis on ensuring that its projects going forward would be more rigorously designed. Of the 14 projects that SSA has initiated since 1998, 13 were early enough in the planning or design stages at that time to give SSA an opportunity to make such improvements. Since that time, SSA has completed much of the design work for its 14 projects and provided us with detailed design information for 12 of them, enabling us to assess the rigor of these projects’ designs for our current review.

Our current analysis shows that SSA did use more rigorous methodologies for the projects initiated over the last decade than for its earlier projects. SSA is now using methodologies known as experimental or quasi- experimental designs, which are commonly used by research professionals conducting demonstration projects to estimate the impacts of program or policy changes. On the basis of our assessment, we determined that 11 of the 12 projects’ designs were strong or reasonable when assessed against professional research standards (see table 3). We compared each project’s design against GAO and recognized academic criteria for conducting evaluation research, which were also consistent with statutory requirements that DI projects be generally sufficient in scope and planned in such a way to permit a thorough evaluation of the program or policy changes under consideration. We also determined that the projects currently under way could provide some reliable results if implemented and evaluated as designed.

Despite this progress, we found that SSA did not always meet additional DI and SSI statutory requirements regarding the general applicability of the projects’ results and the use of expert advice, respectively. The authorizing statute for DI demonstration projects requires that the results derived from the projects will obtain generally in the operation of the disability program. While one of the six DI projects, the BOND project, has been designed to yield nationally representative information about the impacts of the project, the statute does not require that the results be applicable to all DI beneficiaries nationwide. However, the results should apply to a larger group of beneficiaries than just those that participated in the demonstration project, and SSA may be able to apply the results from three other DI projects—the Accelerated Benefits demonstration project, the Benefit Offset - 4 State Pilot, and the Mental Health Treatment Study— more generally because it plans to implement and evaluate the projects in a consistent manner at multiple sites. In addition, one of the two jointly authorized projects—the State Partnership Initiative—did not yield generally applicable results because the projects were not implemented consistently across each state. The authorizing statute for SSI projects requires the Commissioner of SSA to obtain the advice and recommendations of specialists who are competent to evaluate the proposed projects as to the soundness of their design, the possibilities of securing productive results, the adequacy of resources to conduct the proposed research or demonstrations, and their relationship to other similar research or demonstrations already completed or in process before entering into a contract, grant, or cooperative agreement for the project. However, SSA obtained advice from experts for only two of the six SSI projects. Finally, SSA generally met other design criteria required by statute for the BOND project (see app. II).

To further improve the demonstration projects’ planning and methodological rigor, SSA has used external research professionals to work with the agency on the design, implementation, or evaluation of 12 of the 14 projects. SSA officials have acknowledged the need for additional expertise to design and implement methodologically rigorous demonstration projects. Thus, SSA has awarded, or planned to award, contracts and cooperative agreements to research consultants and universities with such expertise to evaluate 12 of its 14 projects (see app. III). In nine cases, these researchers also worked on the design or implementation of the projects. For example, for the Accelerated Benefits demonstration project, an SSA research contractor also designed how and where the project would be conducted and managed its implementation so that the data needed for the evaluation it plans to conduct will be available. We also found that SSA and most of these researchers communicated regularly when collaborating on these projects and researchers submitted monthly or quarterly progress reports to SSA, which included information on expenditures, progress, and areas of concern that needed to be addressed.

SSA also appointed new program management in 2007. Since that time, the new management team has conducted an internal review of the 10 demonstration projects that were under way at the time of their appointment. SSA officials told us that all projects underwent a thorough review that was conducted by the Acting Associate Commissioner for Program Development and Research and others with appropriate expertise. Documents we obtained indicate that the review identified the projects’ strengths, weaknesses, and whether they were likely to yield reliable, useful results. For example, SSA considered whether a project’s sample size and site selection were appropriate, if it had been implemented in accordance with its design, and if it faced any challenges that would prevent researchers from conducting a rigorous evaluation of its results. SSA concluded that five of its projects would continue, expecting that they were likely to yield reliable impact information; these five projects are currently in progress. However, the agency did need to make significant changes to strengthen one of the five projects’ designs— the BOND project. At an earlier point in the design phase, SSA expanded the BOND project’s scope to include multiple components in addition to the benefit offset, such as a health benefits package. The new management team subsequently determined that the cost estimates and program complexity associated with several of those components raised questions about the feasibility of implementing the project and significantly scaled back the scope of the study.

For the other five projects the new management was reviewing, SSA determined they faced significant limitations or challenges—such as poorly chosen implementation sites and low participation—that made it highly unlikely for them to obtain reliable results or would have been duplicative of other ongoing research (see table 4). Thus, SSA cancelled those projects in 2007 and 2008. Although the only information SSA has obtained from these projects are some lessons learned from the Pediatric Medical Unit project, the agency projected that it would have spent another $82 million had it not canceled them. Based on the information SSA provided us about the challenges facing the projects and the expected future costs of conducting them, its decisions to cancel the five projects appear to have been data-driven and reasonable.

Further, SSA consolidated its research expertise by merging the Office of Disability and Income Security Programs (DISP) with SSA’s Office of Policy in February 2008, creating the Office of Retirement and Disability Policy. As of June 2008, each office’s research unit remains intact and no formal organizational changes were made to the demonstration program, but agency officials told us that the merger has facilitated communication and strengthened relationships between researchers within the agency. For example, experts from the former Office of Policy’s research unit routinely review and provide input on the demonstration projects’ designs and evaluations.

SSA Continues to Lack Policies, Procedures, and Other Mechanisms to Ensure Projects Yield Reliable Information

While SSA is taking steps to generally improve its demonstration projects’ designs and address specific project limitations, it does not have policies, procedures, and mechanisms to ensure that demonstration projects will yield reliable information about the impacts of the programs they are testing. According to internal control standards in the federal government, federal agencies should have policies, procedures, and mechanisms in place to provide reasonable assurance that a program’s objectives are being achieved. However, we found that, as of August 2008, SSA had not fully implemented the recommendations we made in 2004 to help ensure the effectiveness of the demonstration projects. Specifically, SSA continues to lack a formal, comprehensive, long-term agenda for conducting demonstration an expert panel to review and provide regular input on the design and implementation of demonstration projects from the early stages of a project through its final evaluation; and a formal process for fully considering the potential policy implications of its demonstration projects’ results and fully apprising Congress of the results and their policy implications.

We did find that SSA has developed a limited research agenda for its projects that lacks basic details about the projects, including their objectives, schedules, and costs. The agenda was also developed without broadly consulting key internal and external stakeholders to obtain their input. SSA officials told us that they do not plan to update the agenda to reflect that some of the projects have been cancelled. In contrast, other federal agencies that conduct research have published much more detailed research agendas and update them regularly. For example, the Department of Education’s National Institute on Disability and Rehabilitation Research publishes a 5-year plan that outlines priorities for rehabilitation research, demonstration projects, training and related activities, explains the basis for such activities, and publishes the plan for public comment before submitting it to Congress.

In addition, agency officials said that SSA planned to continue using experts only on an ad hoc basis, citing how potential conflicts of interest could pose challenges to serve on an expert panel as we had recommended. SSA established ad hoc panels or consulted with outside experts for 8 of its 14 demonstration projects, including 4 of those that are currently in progress. Many of SSA’s project officers and contractors reported that this was a positive experience, and SSA management has told us that they plan to continue using experts in the future. However, under this approach SSA may miss the opportunity to obtain advice more broadly on the demonstration program. For example, the panels and experts that SSA used were brought on board after the agency decided to initiate the demonstration projects; therefore, they were not in position at an early enough point in time to help SSA consider whether a demonstration project or an alternative research approach was, in fact, the best way to meet the agency’s needs. Furthermore, SSA did not regularly seek input from the Social Security Advisory Board or the National Council on Disability, which both play key roles in federal disability policy and could be in a position to advise SSA more broadly on the demonstration projects.

In addition, our prior work found that SSA had not sufficiently provided information on the status and results of its demonstration projects to Congress. In our current review, we found that SSA regularly submits annual reports about the DI demonstration projects to its congressional oversight committees. While SSA meets its statutory requirements by submitting these reports, the information in them is generally limited to descriptions of the projects’ objectives and the dates of upcoming milestones. Similarly, the information that SSA reports about its SSI demonstration projects is limited to brief descriptions in the agency’s annual congressional budget justifications. Key information that could help Congress monitor the progress of the demonstration projects— including project costs, potential risks and obstacles to their success, or the policy implications of their results—was rarely included in the annual reports or budget justifications. However, SSA officials also told us that they sometimes share additional information with Congress about the demonstration projects. For example, SSA officials told us they met with congressional committees in October and November 2007 to share information about its design plans for the BOND project.

In addition to not fully addressing our prior recommendations, SSA does not have written policies and procedures governing how it should review and operate its demonstration program. Accordingly, SSA does not have a written policy requiring SSA management to review its project officer’s demonstration projects on a regular basis. Standards for internal control in the federal government state that managers should compare a program’s actual performance against expected targets and analyze significant differences. Although the new program management team reviewed each of the demonstration projects at the time of their appointment, SSA does not have a written policy requiring such a review process periodically throughout the design, implementation, and evaluation phases of each project. SSA’s lack of a policy to systematically review each project on a periodic basis contributed to problems sometimes going undetected after the projects were implemented, and they did not yield the data needed for their evaluations. For example, because SSA was not actively involved in implementing or monitoring the Florida Freedom Initiative demonstration project, it was not in a position to take steps to ensure that the project proceeded as planned, and staff at the state level failed to enroll enough participants to generate data for the evaluation that SSA planned. Therefore, no evaluation was conducted for this project.

In addition, we found that SSA does not have written procedures for its project officers to follow as they design, implement, and evaluate its demonstration projects. Such procedures could be used to ensure that standard research practices, such as conducting pilot phases and including internal and external stakeholders, are applied when planning and implementing the demonstration projects. Specifically, SSA does not require staff to regularly use pilots to test projects’ underlying assumptions, operational logistics, or feasibility before they are implemented. As a result, SSA planned or conducted pilots or phased implementations for only 8 of its 14 projects, although GAO criteria for evaluation research emphasize the importance of conducting pilots, as they are a critical test of a project’s design. At least four of the projects that did not include pilot phases experienced the type of logistical challenges that pilots are intended to identify. For example, the Homeless Outreach Projects and Evaluation demonstration project experienced start-up delays because of compatibility problems between the contractor’s online data collection system and the computer systems at the 41 sites where the project was implemented. If SSA had conducted a pilot phase for this project, it may have detected these issues at a smaller number of sites and developed a plan to resolve them prior to implementing the full project.

In addition, SSA does not have written procedures directing its project officers or contractors to routinely consult with internal and external stakeholders when planning the demonstration projects. We found that at least 11 of the 14 projects experienced challenges or limitations because SSA had not obtained sufficient input from, or coordinated effectively with, internal and external stakeholders. For example, SSA officials told us that they were aware that the Benefit Offset - 4 State Pilot project and the Benefit Offset National Demonstration project required a change to internal SSA processes for calculating DI benefits but did not coordinate with key internal stakeholders early on to determine how to make this change, and no systematic process was put in place. As a result, SSA had to calculate payments by hand for the Benefit Offset - 4 State Pilot, and the BOND project’s implementation has been delayed while SSA now works with its internal stakeholders to determine how to make the needed changes. Furthermore, SSA officials, contractors, and representatives from the various demonstration projects’ implementation sites told us that there was little input from internal stakeholders and that internal coordination problems existed on at least seven of these projects, including three of those that were cancelled. We also found that lack of coordination or communication with external stakeholders led to challenges in at least seven of these projects, including four of those that were cancelled. For example, coordination problems between the two contractors for the California Rise project resulted in the project’s components being designed in isolation from each other, which complicated the evaluation plans and eventually contributed to the project’s cancellation. In addition, SSA did not always include the prospective implementation sites in the planning and design of its projects, although they could have provided insight into the feasibility and logistical requirements of the project.

While SSA has periodically provided direction informally to its project officers, some project officers told us that more formal guidance would have helped them to better understand what steps were necessary and expected, and we concluded from our discussions with others that such guidance would have been helpful. SSA recognizes that the program’s lack of written procedures is a limitation and is drafting a guidebook on standard research practices for staff to follow when planning and designing demonstration projects. Although an SSA official told us that this document is a work in progress, it appears that the handbook will include key procedures for designing a project, such as identifying what data is needed for the evaluation and how it should be obtained. It also directs project officers to assemble a team for project development that includes staff from across the agency who will be able to collaborate and provide the input necessary to address the multiple components of the demonstration project. The draft guidebook also includes provisions for assembling a research panel composed of internal and external experts. This panel would review proposed research projects and identify those that present the most promising opportunities, taking into consideration the extent to which prior research has already addressed the topic. However, in its current form, the guidebook provides little direction for the implementation or evaluation phases of the demonstration projects, and SSA officials had not finalized it as of May 2008.

Without comprehensive written policies and procedures governing how SSA manages and operates its demonstration programs, the project objectives, designs, and evaluation plans may be impacted during times of organizational change. Because government operating conditions continually change, agencies should have mechanisms in place to identify and address any special risks arising from such changes, especially those caused by hiring new personnel to occupy key positions in the agency. However, because SSA lacks mechanisms such as a standing advisory panel or written policies and procedures to provide continuity for its demonstration program when organizational changes occur, it cannot guarantee that institutional knowledge about the projects is shared or that the impacts of such changes are considered as the projects progress. SSA has experienced several organizational changes since the first of these projects was initiated in 1998, which have included the demonstration program’s relocation from the Office of Policy to the Office of Disability and Income Support Programs in 2002, program management’s replacement in 2007, and the Office of Policy and Office of Disability and Income Support Program’s merger in 2008. At least six of SSA’s projects experienced schedule delays and cancellations, in part because newly appointed officials made significant changes to some projects or determined that because others faced significant limitations or potential challenges it was not in the agency’s interests to continue them (see fig. 1). While certain management actions may be reasonable, SSA’s lack of written policies and procedures governing how such steps are taken leaves current and future projects vulnerable to disruption. For example, we found that the Benefit Offset National Demonstration project is still in the design phase after 9 years, during which time it has gone through numerous revisions by different program managers and was moved from one office to another. As of August 2008, an interagency working group was determining how to implement this administratively complex project. SSA has put the project’s implementation and evaluation on hold until this issue has been resolved.

Conclusions

For over two decades, SSA has had the authority to conduct demonstration projects to test strategies that could address the challenges posed by the low rate of return into the workforce and the growing number of applicants and disabled beneficiaries. However, the agency has missed opportunities to identify ways to modernize DI and SSI programs and policies because it has generally not conducted the demonstration projects effectively. Since 1998 alone, SSA has spent over $150 million on 14 demonstration projects; yet these projects have generated limited information about the impacts of the strategies that were being tested. Although many of these projects were generally well designed, SSA’s lack of written polices, procedures, and mechanisms for managing and operating these projects is one of the key reasons the projects SSA has completed and cancelled to date were generally not implemented and evaluated in a way that yielded reliable, data-driven impact information. As a result, Congress, SSA, and other organizations that play a critical role in federal disability policy continue to lack key information about important issues, such as the impact of providing health care or employment supports to DI and SSI beneficiaries as a means to help beneficiaries achieve self sufficiency and leave the rolls.

SSA’s five demonstration projects currently in progress have the potential to identify solutions to some of SSA’s challenges. However, if SSA does not address the limitations in the way it manages and operates demonstration projects, these projects may encounter the same challenges that past projects have faced, and SSA could again have little to show for its efforts. Given that SSA estimates it will spend approximately $220 million over the next several years to complete these projects, it is important that steps be taken to make the projects less vulnerable to the challenges and organizational changes they could encounter in the future. SSA’s actions to review its demonstration projects and to begin drafting guidance to help staff better plan and design its projects are encouraging first steps. As SSA officials work toward finalizing this guidance, it is also necessary for the agency to address its lack of written policies and procedures for managing and operating its projects during their implementation and evaluation phases. SSA should also take action to fully implement the recommendations we made in 2004. Implementing those recommendations by fully developing its research agenda, establishing an expert panel to advise it about the projects on a regular basis, and improving its communications with Congress could help improve the effectiveness and transparency of its demonstration program going forward.

Recommendation for Executive Action

To improve SSA’s management of its demonstration projects we recommend that the Commissioner of Social Security direct the Deputy Commissioner for the Office of Retirement and Disability Policy to establish written policies, procedures, and mechanisms for managing and operating its demonstration projects that are consistent with standard research practices and internal control standards in the federal government, including those for coordinating with internal and external stakeholders and sharing information with Congress.

Agency Comments and Our Evaluation

We obtained written comments on a draft of this report from SSA, which are reproduced in appendix IV. We incorporated technical comments we received throughout the report, as appropriate.

In response to our draft report, SSA generally agreed with our recommendation and acknowledged the need to develop a guidebook to assist its staff in the design, implementation, and evaluation phases of its demonstration projects. SSA further discussed its existing processes and written procedures for managing and reviewing its programs, including the demonstration project program. While noteworthy, we continue to believe SSA needs to establish written procedures that incorporate professional research standards and internal control mechanisms for ensuring that the demonstration projects yield reliable information about their impacts. SSA considers its current guidebook a work in progress. Further, SSA stated that the agency has taken steps in recent years to address our prior recommendations. While we acknowledge SSA’s efforts, the agency needs to take additional steps to fully implement them. For example, SSA continues to lack a standing expert panel to review and provide regular input on the demonstration projects overall, even though it has employed subject matter experts for some of its demonstration projects. Although SSA officials have raised concerns about the difficulty of establishing an expert panel because research contractors serving on the panel would be precluded from working on individual projects, we continue to believe that such a panel could be established. As previously recommended, this panel should also include SSA’s key research personnel and outside disability experts in addition to researchers.

We are sending copies of this report to the Commissioner of SSA and other interested parties. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions concerning this report, please contact me at (202) 512-7215. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Appendix I: Scope and Methodology

To determine how SSA has used its current demonstration authority, we reviewed legislation authorizing the Social Security Administration (SSA) to conduct Disability Insurance (DI) and Supplemental Security Income (SSI) demonstration projects, prior GAO report and SSA’s Office of the Inspector General (OIG) reports, and reports to Congress on the demonstration authority. We reviewed documents from SSA and from publicly available sources, including the Federal Register and reports by other research organizations. We interviewed current and former SSA officials in the Office of Disability and Income Security Programs, specifically, the Office of Program Development and Research (OPDR) and the Office of Research, Evaluation and Statistics who had responsibility for, or involvement in, the demonstration projects. We also interviewed research contractors that worked on the demonstration projects and individuals from organizations that have a key role in federal disability policy. In addition, we interviewed staff from sites where SSA implemented 9 of the 14 demonstration projects. We selected sites that included ongoing, cancelled and completed projects, and represented diverse geographic regions throughout the United States.

To better understand SSA’s DI and SSI demonstration projects, we reviewed SSA documents describing the purpose, design, and status for all demonstration projects that were in progress, completed, or had been cancelled prior to completion. These documents included requests for proposals, project plans and schedules, interim or final project reports, and reports to Congress from 1998 to 2008. We used these documents to identify key characteristics of the projects, including the policy issues addressed, use of contractors, the authority used to conduct each project, project timelines, and information resulting from each project. We also examined the issues SSA tested or was in the process of testing in its demonstration program. We reviewed the authorizing statutes for the DI and SSI demonstration programs, as well as requirements for specific demonstrations included in the Ticket to Work and Work Incentives Improvement Act of 1999 to determine the extent to which the projects in SSA’s demonstration program address statutory requirements. For projects cancelled during this time period, we collected cancellation memos and other documentation to determine SSA’s reasons for the cancellations.

To describe the costs associated with the program, we collected expenditures data from SSA for each project—including funds spent to date for each project and total anticipated funding for the projects that are currently in progress. We assessed the reliability of the budget data by (1) manually checking the required data elements, (2) reviewing existing information about the data and the system that produced them, and (3) interviewing agency officials knowledgeable about the data. When we found discrepancies, we brought them to the attention of SSA officials and worked with them to correct the discrepancies before conducting our analyses. Based on these efforts, we determined that the data were sufficiently reliable for the purposes of this report.

To assess the extent to which SSA’s demonstration projects were designed in accordance with professional research standards and statutory criteria, we reviewed the most current information that SSA provided about each project, either an evaluation design or final evaluation report. These design and evaluation methodologies were assessed against professional research standards, consistent with the authorizing statutes’ methodological requirements and GAO’s and recognized academic criteria for conducting evaluation research. Key components of the professional research standards include methodological rigor of the project’s design and evaluation and their appropriateness given the purpose of the research (e.g. use of an experimental or quasi-experimental design for an impact evaluation); appropriate handling of any problems encountered when implementing the evaluation’s design, such as participant attrition or insufficient sample sizes; appropriate handling of any problems encountered with the data, such as missing values or variables; appropriate variables to ensure internal and external validity, given the evaluation’s design; appropriate data analysis and statistical models, such as frequencies or multivariate analysis, given the evaluation’s design; and overall strength of the evaluation design and analysis.

To assess the appropriateness of each study’s methodology for answering the research questions, we developed two data collection instruments based on these professional research standards—one for evaluation designs and one for the final evaluation reports. We then examined the strengths and weaknesses of the evaluation designs and final reports, taking into consideration the project’s objectives, resource constraints, methodological approach, technical adequacy of plans for data collection and analysis, and when available, the presentation of the findings. A social scientist read and coded each evaluation design or final report. A second social scientist reviewed each completed data collection instrument and the relevant documentation to verify the accuracy of every coded item. For each DI demonstration project, we also reviewed the reports to ascertain whether they met statutory requirements that the project’s results be broadly applicable to relevant segments of the DI beneficiary population, not just the project participants. For each SSI demonstration project, we also interviewed agency officials to determine whether SSA met its statutory obligation to obtain the advice and recommendations of specialists who are competent to evaluate the projects as to the soundness of their design, the possibilities of securing productive results, the adequacy of resources to conduct them, and their relationship to other similar research or demonstrations already in progress. We also identified key provisions of the demonstration authority statutes to assess SSA’s compliance with congressional reporting requirements.

To address SSA’s planning and management of its demonstration projects, we interviewed SSA management and staff about the agency’s policies, guidance and processes on developing and implementing demonstration projects, and collected supporting documentation where available. We assessed the adequacy of SSA’s internal controls using the criteria in GAO’s Standards for Internal Control in the Federal Government, GAO/AIMD 00-21.3.1, dated November 1999. These standards, issued pursuant to the requirements of the Federal Managers’ Financial Integrity Act of 1982 (FMFIA), provide the overall framework for establishing and maintaining internal control in the federal government. Also pursuant to FMFIA, the Office of Management and Budget issued Circular A-123, revised December 21, 2004, to provide the specific requirements for assessing the reporting on internal controls. Internal control standards and the definition of internal control in Circular A-123 are based on the GAO Standards for Internal Control in the Federal Government.

Appendix II: Additional Statutory Criteria for the Benefit Offset National Demonstration Project

In addition to meeting professional research standards and the mandated methodological requirements for the DI demonstration projects, the Benefit Offset National Demonstration (BOND) project is required to address a number of specific issues. We found that BOND met all of these requirements except the one to test the project’s effect on induced entry to and reduced exit from the DI rolls (table 5).

In addition, while not included as part of the BOND demonstration, SSA plans to conduct research on variations in the amount of the offset as a proportion of earned income to determine the appropriate offset disregard, in accordance with the DI authorizing statute’s requirements.

Appendix III: SSA’s Use of Research Contractors and Universities on the Demonstration Projects

Research consultant or university

Projects conducted under DI demonstration authority (prime contractor)

Mathematica Policy Research (subcontractor)

SSA awarded MDRC a contract in 2006 for the project’s design, implementation, and evaluation. MDRC’s design for the project and evaluation expanded on preliminary design criteria SSA developed between 2004 and 2005, and published in a Request for Proposal. MDRC finalized the design in collaboration with Mathematica Policy Research and SSA. MDRC and Mathematica began enrolling participants in the project in October 2007. Implementation will continue through 2011. MDRC is conducting the impact evaluation and Mathematica is conducting the process evaluation. SSA expects the evaluation to be completed in 2011.

SSA awarded Abt Associates a contract in 2004 for the project’s design and will award Abt a second contract for its implementation and evaluation contingent on successful completion of the design. Abt’s design for the project and evaluation is based on preliminary design criteria SSA developed between 1999 and 2004 and published in a Request for Proposal. Abt is finalizing the design and planning for implementation in collaboration with SSA.

SSA awarded Mathematica Policy Research a contract in 2006 to design and implement the health services part of the project and to design and conduct the evaluation. Mathematica’s design for the heath services component and the evaluation is based on preliminary design criteria SSA developed between 2004 and 2006 and published in a Request for Proposal. Mathematica worked on the design in 2006 and 2007, and SSA reviewed and provided input on it. Mathematica expected to implement the project in 2007 and complete its evaluation in 2011, but SSA cancelled the project in 2007 before implementation began.

Disability Research Institute (Rutgers University)

SSA awarded the Disability Research Institute a contract in 2000 for Rutgers University to design, implement, and evaluate an Early Intervention pilot project. Rutgers proposed a design for the project and evaluation and submitted an evaluation design report for SSA’s approval in 2002. Rutgers planned to implement the project in 2003 and begin its evaluation in 2004, but the project remained under design until the contract expired in 2005.

SSA modified Abt’s contract for the BOND project in 2005 to incorporate Early Intervention into BOND’s design, implementation, and evaluation. Abt proposed options to incorporate the Early Intervention project, but SSA later decided to cancel it and eliminated it from Abt’s contract in 2007.

SSA awarded a contract to Westat in 2005 to implement and evaluate a project SSA designed between 2003 and 2005. Westat began working with the project sites in 2005 to prepare for implementation and began enrolling participants in the project in 2006. Implementation is scheduled to continue through 2010. Westat began the evaluation in 2008 and will complete it in 2011.

DOL awarded a contract to the University of Iowa College of Law, Law Health Policy and Disability Center to provide technical assistance for the project’s implementation and to evaluate the project. SSA provided partial funding for the contract through an Interagency Agreement with DOL in 2002, 2003, and 2004. Implementation began in 2003. The University of Iowa did not evaluate the impact of the project because it could not meet SSA’s data security requirements in order to obtain data needed for the evaluation. SSA officials told us that DOL plans to evaluate the project under a new contract with Mathematica Policy Research, but SSA is not funding that evaluation.

SSA awarded a contract to George Washington University in 2004 to conduct preliminary research related to the Early Identification and Intervention project. SSA designed the demonstration project between 2004 and 2006 and issued a Request for Proposal for its implementation in 2007, but did not award a contract to implement or evaluate the project.

SSA funded a contract for Mathematica to design an evaluation for the project through an Interagency Agreement with the Department of Health and Human Services in 2004. Due to low enrollment, data needed for the evaluation was not collected by the State of Florida, which designed and implemented the project between 2003 and 2007, and the evaluation was not conducted.

SSA awarded a contract to Westat in 2004 to evaluate a project SSA designed between 2003 and 2004. SSA implemented the project in 2004. Westat completed its evaluation in 2007.

SSA planned to issue a Request for Proposal to hire a professional research organization to design, implement, and evaluate the project but cancelled it before it awarded the contract.

University Centers on Disabilities (AUCD)

SSA awarded a contract to AUCD in 2006 to design, implement, and evaluate the project; however, AUCD’s design for the project began in 2004 under an extension of another contract. AUCD partially implemented the project in 2006 and 2007. SSA cancelled the project in 2008 before AUCD completed the implementation or evaluation.

Projects jointly authorized under DI and SSI authorities Virginia Commonwealth University (VCU) (prime contractor)

Mathematica Policy Research (subcontractor)

SSA awarded multiple contracts to VCU starting in 1998 to provide technical assistance to the states that were implementing the project and to conduct an evaluation. VCU completed its evaluation in 2006. SSA awarded three contracts executed by VCU’s subcontractor, Mathematica: one in 1998 to provide technical assistance to the states that were implementing the project, another in 1999 to design the evaluation, and one in 2003 to conduct the evaluation. Mathematica completed its evaluation in 2005.

Mathematica Policy Research (prime contractor)

MDRC (subcontractor)

SSA originally designed and implemented the project in 2003, but due to methodological limitations, SSA awarded a contract to Mathematica in 2005 to redesign, implement, and evaluate the project. Mathematica began implementing the new design in 2006. Implementation will continue until 2013. Mathematica will complete its evaluation in 2014.

Appendix IV: Comments from the Social Security Administration

Appendix V: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the individual named above, key contributions to this report were made by Michael Collins, Assistant Director; Jason Holsclaw and Anne Welch, Analysts-in-Charge; Dana Hopings; Annamarie Lopata; Jean McSween. Additional support was provided by Kenneth Bombara; Daniel Concepcion; Jennifer Cook; Cindy Gilbert; Sharon Hermes; Joanie Lofgren; Joel Marus; Mimi Nguyen; Patricia Owens; Daniel Schwimer; Kris Trueblood; Kathy White; Charles Willson; Elizabeth Wood; and Jill Yost.

Related GAO Products

Federal Disability Programs: More Strategic Coordination Could Help Overcome Challenges to Needed Transformation. GAO-08-635. Washington, D.C.: May 20, 2008.

Social Security Disability: Better Planning, Management, and Evaluation Could Help Address Backlogs. GAO-08-40. Washington, D.C.: December 7, 2007.

High Risk Series: An Update. GAO-07-310. Washington, D.C.: January 2007.

Federal Disability Assistance: Wide Array of Programs Needs to be Examined in Light of 21st Century Challenges. GAO-05-626. Washington, D.C.: June 2, 2005.

Social Security Disability: Improved Processes for Planning and Conducting Demonstrations May Help SSA More Effectively Use Its Demonstration Authority. GAO-05-19. Washington, D.C.: November 4, 2004.

SSA’s Rehabilitation Programs. GAO/HEHS-95-253R. Washington, D.C.: September 7, 1995.

Impact of Vocational Rehabilitation Services on the Social Security Disability Insurance (DI) Program. GAO/HRD-T-88-16. Washington, D.C.: May 26, 1988.

Social Security: Little Success Achieved in Rehabilitating Disabled Beneficiaries. GAO/HRD-88-11. Washington, D.C.: December 7, 1987.

Social Security: Observations on Demonstration Interviews with Disability Claimants. GAO/HRD-88-22BR. Washington, D.C.: December 3, 1987.

Social Security: Demonstration Projects Concerning Interviews with Disability Claimants. GAO/HRD-87-35. Washington, D.C.: February 19, 1987.