#### Ticket to Workand Work Incentives Advisory Panel

***Advice Report to the Commissioner of the Social Security Administration:***

***Design Issues Relating to the Adequacy of Incentives Study***

***June 18, 2002***

Table of Contents

Executive Summary1

Message from the Panel7

Summary of A Social Security Administration Report:Working Paper 003—Design of the Study of the Adequacy of Incentives for the Ticket to Work Program9

Summary of Experts Roundtable, July 27, 200111

Summary of Issues15

Questions and Recommendations19

Summary33

Appendices35

1. Ticket To Work And Work Incentives Advisory Panel

2. Design And Evaluation Committee

3. List Of Experts, Experts Roundtable, July 27, 2001

4. Public Commenters, July 27, 2001

5. Individuals Providing Public Comment In Writing    Following July 27, 2001

6. Staff, July 27, 2001

7. Presenters, July 27, 2001

8. References and Bibliography

**Executive Summary**

The purpose of the Panel’s Advice Report is to provide analysis and recommendations regarding design of the Adequacy of Incentives Study authorized by the Ticket to Work and Work Incentives Improvement Act of 1999. The Adequacy of Incentives Study was mandated by Congress to evaluate how the Ticket to Work and Self-Sufficiency Program can be used to increase employment among those with significant disabilities. The Commissioner is required to report to Congress on recommendations for a method or methods to adjust payment rates to Employment Networks to ensure the participation of individuals with disabilities in four specific groups. The Commissioner must implement the necessary adjusted payment rates prior to full implementation of the Ticket to Work and Self-Sufficiency Program. The statute requires that the Panel be consulted during the development and evaluation of the study.

This Advice Report is based on relevant documents, including reports from an Experts Roundtable and public testimony coordinated and convened by the Design and Evaluation Committee of the Work Incentives Advisory Panel in Washington, DC, on July 27, 2001. In addition, this Advice Report is based on related research on employment of people with significant disabilities and targets the four groups in the disability population noted in the Adequacy of Incentives provision of the legislation who are likely to be “involuntary nonparticipants.” These four groups are individuals with a need for ongoing support and services, individuals who need high-cost accommodations, individuals who earn a subminimum wage, and individuals who work and receive partial cash benefits.

Summarized in this Advice Report are a report by the Social Security Administration (Working Paper 003*, Design of the Study of the Adequacy of Incentives for the Ticket to Work Program*, Huynh and O’Leary, July 14, 2001); key points from the July 27, 2001, Experts Roundtable; and Panel and Experts Roundtable recommendations that will provide a framework for additional research related to the Adequacy of Incentives. The points offered in this Advice Report are intended to support and enhance, not supplant, the overall proposal described by the Social Security Administration’s Office of Policy in Working Paper 003 and discussed at the Experts Roundtable.

**Panel Recommendations**

The 13 recommendations offered here include recommendations from the Panel related to the ongoing process for the Adequacy of the Incentives Study. They are followed by recommendations from the Experts Roundtable on nine specific issues raised by the Social Security Administration (SSA) and/or the Panel. 

**Design Process of the Study**

**Panel Recommendation 1:****An ongoing structure must be established for review of emerging information, analysis of trends, and creation of recommendations regarding modification of incentives and making use of multiple evaluation methods. 

**Panel Recommendation 2:****The Social Security Administration should convene an Adequacy of Incentives Advisory Team, similar to the team of consultants approach being used under the $1 for $2 research, for ongoing technical support for the Adequacy of Incentives Study, specifically related to the four targeted groups. This group should consist of diverse individuals who have research background, knowledge, and/or expertise in the successful employment of individuals in the four groups. This group could include, but not be limited to, the experts identified by the Panel for the Adequacy of Incentives Expert Roundtable. These experts would be charged to further develop recommendations from the Experts Roundtable for serious consideration by the Social Security Administration. At least two members of the Adequacy of Incentives Advisory Team should become members of the Technical Evaluation Support Group, the technical group advising Social Security Administration on overall Ticket evaluation, to ensure coordination of the two research efforts. 

**Panel Recommendation 3:****Given the impact of this and other studies the Office of Policy conducts on the lives of people with disabilities and their families, the Office of Policy should develop a policy and practice for incorporating the input and experiences of consumers, their families, and/or their representatives as it develops new research designs, including the Adequacy of Incentives Study. 

**Panel Recommendation 4:****As the Ticket to Work and Self-Sufficiency Program is implemented, the Social Security Administration should provide to the Panel, Congress, and the President ongoing interim reporting on the participation of members of the four targeted groups in the Ticket to Work and Self-Sufficiency Program. 

**Panel Recommendation 5:****The Panel recommends that Congress approve a technical amendment to require the Social Security Administration to provide to the Panel, Congress, and the President ongoing interim reporting on the participation of members of the four targeted groups in the Ticket to Work and Self-Sufficiency Program. 

**Panel Recommendation 6:****Ongoing data collected and outcomes determined for the Adequacy of Incentives Study should be included in the Social Security Administration’s overall Ticket evaluation research.

**Panel Recommendation 7:****Due to delays in issuing the tickets, the deadline for the Adequacy of Incentives Report to Congress should be extended by the amount of time the ticket rollout has been delayed (13 months). The Panel recommends that Congress approve an amendment to adjust the timeline for the Adequacy of Incentives Report so that it is due 49 months, rather than 36 months, after the enactment date. Implementation of the necessary adjusted payment rates should occur as soon as practical following the release of the Adequacy of Incentives Report but no later than 1 year after its release. Further, the Panel requests preliminary progress reports on the Adequacy of Incentives Study at 36 and 42 months after the date of enactment of the law. 

**Identifying the Four Groups**

**Panel Recommendation 8:** While there is a need to take advantage of what is known about the diagnostic categories with high proportions of those most likely to fall into one of the four groups relative to work capability, the Social Security Administration, in addition to diagnostic data, should use other information, such as survey information, information on earnings, onset date, and medical improvement assessments. SSA should also go to outside data sources, such as those identified in Expert Roundtable Recommendation 1 (below), to identify members of the four target groups.

**Survey Methodology**

**Panel Recommendation 9:** The Social Security Administration has proposed a survey to supplement its administrative data, improving its ability to target the four groups. If the survey uses a telephone protocol, the protocol should include two other elements. First, there should be face-to-face home interviews if the phone survey fails for a given respondent, except for cases of incorrect phone numbers, empty residences, and so on. Second, if the potential respondent’s disability prevents effective participation in the survey, the interviewer should use as a proxy another individual who knows the respondent well and, preferably, who has communicated on the respondent’s behalf in the past, such as a parent, spouse, sibling, or supervisor in a group home. Members of the Adequacy of Incentives Advisory Team (see Panel Recommendation 2) should review the interview protocol used by the contractor conducting the survey.

**Supplementary Research**

**Panel Recommendation 10:** Recognizing the need for research to supplement what can be studied using Social Security administrative data, SSA has proposed a program of supplementary research. It has solicited the suggestions of the Panel for possible research topics to supplement the research using SSA data. The Panel recommends the following:

· Review existing research on the four groups or related groups, including relevant behavioral demonstrations and field experiments. 

· Conduct or commission new research on these groups using State data or other Federal data.

· Conduct or commission new research related to Employment Networks and other providers.

· Review and consider best practices of successful programs relating to employment of those with significant disabilities.

· Review best practice systems of results-based funding and/or individual milestones payments.

· Conduct or commission a review of supported employment literature, research, and best practices.

Conduct or commission ongoing demonstration projects, modeling, and evaluation of the effectiveness of alternative incentives identified.

**Employment Networks**

**Panel Recommendation 11:** SSA should identify the data elements that Employment Networks (ENs) should collect on whom they serve and do not serve from the four functional groups identified in the statute. The data should include descriptions of why a ticket is accepted or is not accepted on the basis of the EN’s perception of cost, degree of difficulty, or lack of expertise on the part of the EN to provide the necessary services. SSA should use a sampling of ENs rather than requiring all ENs to collect the data. SSA should pay the ENs to offset the administrative burden of management information system design.

**Panel Recommendation 12:****SSA should conduct or commission qualitative research, including interviews and focus groups as appropriate, to assess the adequacy of current incentives to large and small providers, including community rehabilitation programs. Information and recommendations should be collected regarding the amount and timing of payments and other factors that affect potential participation in the Ticket Program and integrated community employment at minimum wage or higher levels for individuals in the four groups.

**Various and Diverse Populations**

**Panel Recommendation 13:** SSA should ensure that the data collected include attention to national demographics such as rural and urban settings, minority representation, and socioeconomic conditions across the country.

**Recommendations From the Experts Roundtable**

The Panel convened an Experts Roundtable on July 27, 2001, as part of its effort to bring expertise to bear in advising SSA on the design of the Adequacy of Incentives Study. Following are the recommendations that emerged from the Experts Roundtable in response to the nine questions posed by the Social Security Administration and the Panel. The Panel strongly urges the Social Security Administration to seriously consider and further develop these recommendations in designing research related to the Adequacy of Incentives Study and its context within the broader evaluation of the Ticket to Work and Self-Sufficiency Program. 

**Identifying the Four Groups**

**Roundtable Recommendation 1:** In addition to the administrative data sets SSA has proposed using, use appropriate Federal databases (e.g., Office of Special Education and Rehabilitative Services, Rehabilitation Services Administration, and Department of Labor); State databases (e.g., State developmental disability agencies and State mental health agencies); data from providers of day treatment, day activity programs, sheltered workshops, and supported employment programs (e.g., data from the Employment Intervention Demonstration Program funded by the Center for Mental Health Services); and existing research studies in order to identify and gather information from and about members of the four groups. In addition, the Helen Keller National Center has data on 7,000 persons who are deaf and blind, some of whom are seeking employment.

**Roundtable Recommendation 2:** Oversample potential members of these four groups who can be located in the administrative data from Social Security, Office of Special Education and Rehabilitative Services, Rehabilitation Services Administration, Medicaid, the Department of Labor, and/or other Federal sources.

**Outcomes of Interest**

**Roundtable Recommendation 3:** Track basic employment outcomes (jobs, wages, hours, choice, etc.) for members of the four groups once identified.

**Roundtable Recommendation 4:** Gather information about access to and use of the Ticket, with particular attention to whether nonparticipation is voluntary or involuntary. 

**Roundtable Recommendation 5:** Conduct or commission qualitative research, including interviews and focus groups, to gather more extensive information about participation, economic and other benefits, and ways to attain improved incentives.

**Roundtable Recommendation 6:** Analyze the data collected related to patterns of outcomes, cost/benefit, participation, and modifications of incentives for members of these four groups compared with others making use of the ticket.

**Maximizing Outcomes**

**Roundtable Recommendation 7:** The Social Security Administration should ensure that studies of these four groups include attention to the issue of the “mix” of funding, government programs, and other incentives and disincentives that affect the employment of people with disabilities. These studies should include analysis of how various funding streams can be used together to enhance incentives to Employment Networks and providers and result in employment for people in the four groups.

**Successful Efforts to Employ People With Significant Disabilities in Community Employment at Minimum Wage Levels or Higher**

**Roundtable Recommendation 8:****The Social Security Administration should study the successes of supported employment programs, Rehabilitation Services Administration Choice projects, and other demonstration projects of the past decade for members of these four groups.

**Other Methods to Identify Members of the Four Groups**

**Roundtable Recommendation 9:****The Social Security Administration should consult other databases to identify members of the four groups. Such databases should include other existing national samples as well as State databases for persons with developmental disabilities, persons with mental illness, persons with deafness and blindness, persons with brain injuries, and persons with other disabilities.

**Individualized Milestone Payments**

**Roundtable Recommendation 10:****The Social Security Administration should study the current and recent experiences of States that have begun using results-based funding or some form of payment for individualized milestones toward employment.

**Roundtable Recommendation 11:****The Social Security Administration should conduct or commission cost/benefit analyses and create demonstration projects on modifying the incentives to Employment Networks and providers of services, showing how members of the four groups can benefit from the Ticket and from employment that partially reduces benefits.

**Various and Diverse Populations**

**Roundtable Recommendation 12:** The Social Security Administration should ensure that the data collected include attention to national demographics such as rural and urban settings, minority representation, and socioeconomic conditions across the country.

**Including Consumers, Family Members, and/or Their Representatives in the Study Design and Implementation**

**Roundtable Recommendation 13:****Given the impact of this and other studies the Office of Policy conducts on the lives of people with disabilities and their families, the Office of Policy should develop a policy and practice for incorporating the input and experiences of consumers, their families, and/or their representatives as it develops new research designs, including the Adequacy of Incentives Study. 

**Including Employment Networks in the Study Design and Implementation**

**Roundtable Recommendation 14:** The Social Security Administration should gather data related to the costs to Employment Networks and providers of services and the timeframes for providing access to the Ticket and employment for members of these four groups.

**Roundtable Recommendation 15:** The Social Security Administration should study the relationship between perceived disincentives and actual incentives for Employment Networks and providers of services, including incentives for small providers of services.

**Message from the Panel**

In enacting the Ticket to Work and Work Incentives Improvement Act of 1999, Congress recognized that the Ticket to Work and Self-Sufficiency Program would not be equally available to all individuals with disabilities. In particular, some people with significant disabilities might be involuntary nonparticipants. That is, they might be willing to work but, because of their need for ongoing support and services, high-cost accommodations, limited wages, and/or partial cash benefits, as well as outside factors, they might not find an Employment Network willing or able to take their ticket under the existing financial incentives of the Ticket to Work and Self-Sufficiency Program.

To address this equity issue, Congress included a requirement in the statute that the Social Security Administration identify and implement a payment system that would allow this population to participate in the Ticket Program. The Commissioner is required to report to Congress on recommendations for a method or methods of adjusting payment rates to Employment Networks to ensure equitable participation for individuals in four specific groups. The Commissioner must implement the necessary adjusted payment rates prior to full implementation of the Ticket to Work and Self-Sufficiency Program. The statute requires that SSA consult with the Panel during the development and evaluation of the study—commonly referred to as the Adequacy of Incentives (AOI) Study. The statutory language follows:

*Title I, Sec.101 (h)(C) REPORT ON THE ADEQUACY OF INCENTIVES—The Commissioner shall submit to the Congress not later than 36 months after the date of the enactment of the Ticket to Work and Work Incentives Improvement Act of 1999 a report with recommendations for a method or methods to adjust payment rates under subparagraphs (A) and (B), that would ensure adequate incentives for the provision of services by employment networks of—(i) individuals with a need for ongoing support and services;(ii) individuals with a need for high-cost accommodations;(iii) individuals who earn a subminimum wage; and(iv) individuals who work and receive partial cash benefits.*

*The Commissioner shall consult with the Ticket to Work and Work Incentives Advisory Panel established under section 101(f) of the Ticket to Work and Work Incentives Improvement Act of 1999 during the development and evaluation of the study. The Commissioner shall implement the necessary adjusted payment rates prior to full implementation of the Ticket to Work and Self-Sufficiency Program.*

The Panel has carefully considered issues related to equitable participation in the Ticket Program. This has included consultation with the Social Security Administration, other Federal agencies, and various experts; opportunities for public comment; and Panel deliberations. The Panel’s Design and Evaluation Committee convened an Experts Roundtable of nationally recognized experts, Social Security Administration officials, and Panel members on July 27, 2001. The Roundtable addressed the many complex issues involving the employment of individuals with significant disabilities. The impact of the overall Ticket evaluation on individuals in the four groups and the design of the AOI Study were carefully considered. Subsequently, a well-known expert in the field, Dr. David Mank, assisted the committee and Panel in drafting this report.

This Advice Report represents the Panel’s effort to bring expertise to bear in advising the Social Security Administration on these complex issues and the design of the AOI Study. It became evident during this process that ongoing consideration and advice on this important topic will be necessary. The Panel looks forward to continuing to address these issues and provide advice as the Ticket is implemented and research goes forward.

Summary of A Social Security Administration Report:

*Working Paper 003—Design of the Study of the Adequacy of Incentives for the Ticket to Work Program*

An overall design has been crafted by the Social Security Administration for the Adequacy of Incentives Study that is based in part on Federal databases and includes a process evaluation and an impact evaluation (Huynh and O’Leary, 2001). Such research is required by the Ticket to Work and Self-Sufficiency Program. Specifically, the Social Security Administration is to assess the Adequacy of Incentives structure in serving members of groups considered unlikely to participate and to develop and test alternative incentives as necessary. The Act targets four groups that are considered likely to be “involuntary nonparticipants.” These four groups are individuals with a need for ongoing support and services, individuals who need high-cost accommodations, individuals who earn a subminimum wage, and individuals who work and receive partial cash benefits.

It is important that the Adequacy of Incentives design include specific attention to these four groups from the beginning of the Ticket to Work program evaluation process. As Huynh and O’Leary say, “ ... it enables us to ask: What data do we *want* to have when we get to the point of implementing the evaluation, rather than what do we *have* at that point” (Huynh and O’Leary, 2001, p. 2). 

Working Paper 003 asks two important questions relative to this Advice Report: Are incentives adequate for the target groups? And, if not, how can incentives be modified to better serve these groups? The paper goes on to describe the proposed process for the study of the Adequacy of Incentives. 

Data for addressing the Adequacy of Incentives will be drawn from administrative data in the Social Security Administration and from Internal Revenue Service tax files and possibly Center for Medicaid and Medicare Services (formerly known as the Health Care Finance Administration) data and Rehabilitation Services Administration data. The intention here is to “compare the characteristics of those served in the Ticket to Work program with the characteristics of the SSDI [Social Security Disability Insurance] and SSI [Supplemental Security Income] disabled populations.” (p. 4) The authors also note that while the administrative data will be useful, they are not likely to include important information about ongoing publicly funded supports, accommodations, and hourly wages earned. Because of the lack of some needed data in the administrative data sets, the paper also proposed a survey that would ask beneficiaries about work, wages, access to the Ticket, access to vocational rehabilitation services, and other questions. A phone survey is planned to attempt to collect these data, supplemented by home interviews when information cannot be collected over the phone. Issues with this approach are also discussed, including sampling issues and survey subjectivity. For the purpose of determining how to modify the incentives of the Ticket to Work, the authors propose a theoretical modeling approach because, they argue, “testing alternative incentives is ... not currently feasible given the limited information ... on alternative incentives to test.” (p. 8) The authors expect the Social Security Administration administrative data to be useful, but they note that its usefulness may be somewhat limited. In addition, the survey noted earlier will be designed to help address the issue of determining how to modify Ticket to Work and Self-Sufficiency Program incentives. Issues of subjectivity and defining eligibility are also noted and discussed related to possible modification of these incentives. 

Working Paper 003 also indicates that additional input is needed on three specific questions: What are objective means of identifying the groups? What are the outcomes of interest? What array of services should be considered to maximize outcomes to members of these four groups?

Summary of Experts Roundtable, July 27, 2001

An Experts Roundtable was convened by the Design and Evaluation Committee of the Work Incentives Advisory Panel on July 27, 2001, in Washington, DC. The Panel assembled experts in several fields, with the main focus being efforts to employ those with significant disabilities. Lists of experts in attendance and of those providing public comment are provided in the appendixes to this Report. The meeting provided the opportunity for review and discussion of the design of the study on the adequacy of incentives for individuals in the four groups noted. After presentation of the study design by the Office of Policy and its consultants, the experts offered comments in response to a series of questions posed by both the Social Security Administration and the Work Incentives Advisory Panel. Members of the public also had an opportunity to share their thoughts and impressions on this topic through testimony. 

While the overall design may represent important methods to begin to assess the use of the Ticket to Work and Self-Sufficiency Program and the Adequacy of Incentives, the following points summarize the concerns and input of the Experts Roundtable, public testimony, and testimony received in writing after July 27, 2001:

**Expert Comments**

· The design should be formative in nature, immediately seeking ways to modify and test incentives rather than waiting for a period of time to conduct a summative evaluation to determine whether the design of incentives was adequate. The design should emphasize the question, What will it take for the incentives to be adequate for Employment Networks to serve members of the four groups?

· There is a need for qualitative studies (in-person interviews and focus groups) in addition to quantitative studies.

· There is a need for specific attention to collecting data regarding the issue of incentives to, and perceptions of, providers of services.

· There is concern that the phone survey protocol, even supplemented by home interviews, will be inadequate because of communication barriers for some members of the four groups and lack of understanding of the nature of the Ticket to Work and Self-Sufficiency Program.

· There is concern that the survey should be sensitive to minority and cultural issues as well as to geographical differences.

· There is a need to use existing databases from supported employment data, State data from developmental disability agencies, and State data from mental health agencies.

· There is ample evidence, based on the supported employment initiative, Rehabilitation Services Administration choice projects, and other projects, that individuals in these groups can work productively, although often part time.

· Demonstration projects with members of the four groups are essential to show how the incentives could be modified to work with members of the four groups.

· Evaluation of the Adequacy of Incentives should make use of the experience in States using results-based funding or milestone payments in the vocational rehabilitation system.

The evaluation design of the Adequacy of Incentives should be considered in the context of the other benefits and services received by members of the four groups.

**Public Comments**

· The AOI Study is the only hope that persons with high support needs have for utilization of their tickets.

· SSA must look beyond its own data banks to employment researchers who have spent years studying supported employment and the successes and the impact work has had on the lives of persons with high support needs.

· Everyone needs to have the opportunity to work, regardless of disability. No one can be left behind.

· The Ticket to Work Program is an extremely exciting opportunity for people with disabilities to gain employment while being in control of funding. It actualizes their role as customers, changing the perceptions and focus of the provider community and the general public. People with more severe disabilities may not be able to benefit from the Ticket to Work Program as it is currently designed. SSA and the Panel must analyze the types of supports needed by these individuals, the associated expenses, and the impact on the person’s life style.

· Because of the multitude of issues that surround individuals with significant disabilities, it is incumbent upon this process to take a broad view of both cost and benefit. Taking a narrow, econometric approach to the study may not reveal an effective payment strategy for these individuals—reducing potential outcomes that might be experienced otherwise.

· A broad-based study will decrease reliance on Social Security and other benefits.

· Individuals with disabilities can become employed and be successful with proper assistance. Without these intensive, individualized supports, though, these individuals are destined to remain on Social Security benefits and other public assistance programs.

· In order to prevent failure of this program, input about the actual cost of supports needs to come from agencies that are proficient in serving individuals with disabilities, and *not*determined in a vacuum or through sterile research methodologies.

· Expertise of people with disabilities, their families, and agency practitioners must be used in the research design.

· SSA and the Panel must do nothing less than draw on the best information in the field to access the answers SSA needs to design payment strategies that will work for all persons who have a disability—especially those who have the most severe disabilities. This cannot be an academic debate about competing research methodologies; it should be about determining where the answers lie and designing a study that will lead to those answers.

· The AOI Study potentially opens another avenue that is desperately needed to open employment for people with significant disabilities.

In Ohio, only 8 percent of people with severe disabilities are employed. In the proposed Ticket regulations, there is little incentive for providers to accept tickets from individuals who need intense and/or expensive supports. The AOI Study is needed to determine alternative payment strategies so these people can use their tickets and become employed with the needed supports.

**Summary of Issues**

The following represents a synthesis of the points derived from the Experts Roundtable, public testimony, literature review, and Design and Evaluation Committee discussions and provides the conceptual underpinning for the Panel and Expert Roundtable recommendations made in this Report. 

1) Individuals in these four groups are less likely to participate in the Ticket Program because they are less likely to stop receiving Social Security benefits altogether.

2) Individuals in these four groups will be extremely difficult to identify in Federal databases because data on relevant dimensions defining these four groups are not directly collected in Federal databases. Some Federal data may allow inferences for some of the members of some of these groups (e.g., there may be relevant data from the Department of Labor regarding individuals earning subminimum wages; there may be relevant data on subsets of the four groups in Medicaid data and in data from the vocational rehabilitation system).

3) It is likely that individuals in these four groups will require a greater degree of effort and long-term support for successful employment and may not be able to work enough hours or earn enough money to eliminate Social Security benefits (e.g., Mank, 1994; Wehman, 2001).

4) Phone survey methodologies will be of limited utility for members of these four groups because of intellectual or other developmental disability, or communication difficulties based on disability, in addition to the issues of English as a second language and minority populations (including Native Americans), which will also affect the broader evaluation design (HSRI, 1999). 

5) There is a need for a “formative” evaluation approach****and other research and evaluation designs for these four groups (which asks the question “Under what circumstances can and do members of these four groups participate?”), in addition to a summative evaluation about the extent to which members of these groups participate (Campbell and Stanley, 1963; Weiss, 1972).

6) There will be major ongoing costs to Social Security as a result of widespread nonparticipation of people in these groups. Individuals in these groups are likely to receive a lifetime of benefits if they do not participate in employment (e.g., Braddock, 2001).

7) There is widespread evidence that individuals who are part of these four groups can work successfully (although often part time) with individualized support structures (supported employment research and evaluation studies, Rehabilitation Services Administration “Choice” projects, etc.). These data are extremely relevant in considering the employment rates and potential of people in the four groups and in determining what Employment Networks need to assist individuals with significant disabilities to be employed (e.g., Mank, Cioffi, and Yovanoff, 1999; Mank, O’Neill, and Jensen, 1998; Revell, Inge, Mank, and Wehman, 1999; Sowers and Powers, 1991; Wehman, 2001).

8) The research proposed by the Social Security Administration on the Adequacy of Incentives should be supplemented in part because Social Security administrative data do not include all the information needed to identify the four groups and in part because of communication issues associated with efforts to survey these groups. Promising sources of information include existing State and local databases as well as efforts to more directly collect new data relative to the individuals in these groups in addition to the adequacy of the incentives to the Employment Networks and providers of employment services (Campbell and Stanley, 1963; Weiss, 1972).

9) The methodologies of the evaluation design must be considered in the complex context of a range of factors that affect the extent to which people with disabilities work and the extent to which the incentives in the Ticket Program are adequate. Additional factors that have an impact on the employment of people with disabilities include benefits management, provider availability and capacity in local areas, and vocational rehabilitation initiatives (Novak, Mank, Revell, and O’Brien, 1999; Wehman, 2001). The nature and funding, including milestone payments, of vocational rehabilitation services, long-term day services, and employment support services by State developmental disabilities agencies or State mental health agencies will also have an impact on employment. The nature of States’ Medicaid Waiver programs and Medicaid Buy-In will influence employment. In addition, the extent to which long-term employment supports and services are funded or not funded in a given State will have an impact (e.g., Braddock, 2001; Butterworth, Gilmore, Kiernan, and Schalock, 1999).

10) “Purposeful” sampling will be useful for qualitative research and focus groups (Guba, 1978; McMillan and Schumacher, 1984). This strategy should include individuals in the four groups as well as Employment Networks and providers. The overall design proposed provides the broad structure for the large quantitative study of the Adequacy of Incentives. The four groups noted present specific challenges for the Adequacy of Incentives and for the study design. State samples provide a method for more closely identifying members of the four groups and for collecting more detailed information. In addition, they provide a context for creating studies specific to Employment Networks and providers in the specific context of the samples of individuals in the four groups. 

11) There is a need to ensure that all methods used are designed with attention to geography, demographics, and economic conditions, as well as the representation and participation of Native Americans and other minority groups. 

12) While the limits of the available administrative data have been noted, it must also be noted that available State databases and databases from national samples have limits related to the completeness and representativeness of the data. Thus, a combination of research, evaluation, and database methodologies is needed to create the most complete (albeit inherently imperfect) picture possible of the participation of members of these four groups and the modification of incentives.

13) While each of the research and evaluation strategies discussed in this Report has limits, it must also be recognized that the combined strategies discussed represent the best possible approach, given the limits, and should be implemented, because the cost of “doing nothing” is failing to provide any useful information about the participation of members of these groups and the modification of incentives.

14) These issues will not be fully resolved in the short run because of the weaknesses in the administrative databases, the need to create alternative evaluation designs for the evaluation of the Adequacy of Incentives, and the complex context of services and funding streams in which services are provided to members of these four groups. This complexity underscores the need for ongoing evaluation and reporting requirements in addition to the Adequacy of Incentives Study, report, and implementation requirements in the statute. 

*Note that the appendices include references as well as a more complete bibliography regarding employment initiatives and research related to likely members of the four groups.*

**Questions and Recommendations**

The analysis and input are considered here in the context of the three primary questions posed by the Social Security Administration (questions 1–3 below) and six additional questions from the Panel (questions 4–9):

15) What are objective means of identifying, through the Social Security Administration administrative data, the target populations as defined in the Ticket to Work and Work Incentives Improvement Act?

16) What are the outcomes of interest?

17) What array of services should be considered to maximize outcomes to members of these four groups?

18) What experts and information resources should be considered regarding successful efforts to employ people with severe disabilities in community employment?

19) What other methods should be used to identify members of the four groups?

20) How can the Social Security Administration demonstrate how individualized milestone payments could be used?

21) How can the Social Security Administration ensure that various and diverse populations are represented in the study?

22) How can consumers, family members, and/or their representatives be included in the study design and implementation?

23) How should Employment Networks be consulted in the Adequacy of Incentives Study?

In addition, the Ticket to Work and Work Incentives Advisory Panel believes that the ongoing process for the design of the Adequacy of Incentives Study is critical to the success of the Ticket to Work and Self-Sufficiency Program for members of these four groups. Thus, recommendations about the ongoing process for study and reporting are defined.

**Panel Recommendations**

**Design Process of the Study**

There are important issues concerning the plan for the evaluation of the Adequacy of Incentives specifically related to the four groups in question. Because it is agreed that the available administrative data and the alternative strategies have limits in providing information about members of the four groups, it will be necessary to define a process for design and implementation of the study that is sensitive to the need for current and ongoing information about participation and about the adequacy of incentives to the Employment Networks and the providers of services. Because the Act mandates attention to the participation of these four groups and to the modification of incentives in the interests of these four groups, the following overarching study design principle recommendation is made:

**Panel Recommendation 1:****An ongoing structure must be established for review of emerging information, analysis of trends, and creation of recommendations regarding modification of incentives and using multiple evaluation methods. 

The following recommendations are intended to support the ongoing evaluation design.

**Panel Recommendation 2:****The Social Security Administration should convene an Adequacy of Incentives Advisory Team, similar to the team of consultants approach being used under the $1 for $2 research, for ongoing technical support for the Adequacy of Incentives Study, specifically related to the four targeted groups. This group should consist of diverse individuals who have research background, knowledge, and/or expertise in the successful employment of individuals in the four groups. This group could include, but not be limited to, the experts identified by the Panel for the Adequacy of Incentives Expert Roundtable. These experts would be charged to further develop recommendations from the Experts Roundtable for serious consideration by the Social Security Administration. At least two members of the Adequacy of Incentives Advisory Team should become members of the Technical Evaluation Support Group, the technical group advising Social Security Administration on overall Ticket evaluation, to ensure coordination of the two research efforts. 

**Panel Recommendation 3:****Given the impact of this and other studies the Office of Policy conducts on the lives of people with disabilities and their families, the Office of Policy should develop a policy and practice for incorporating the input and experiences of consumers, their families, and/or their representatives as it develops new research designs, including the Adequacy of Incentives Study. 

**Panel Recommendation 4:** As the Ticket to Work and Self-Sufficiency Program is implemented, the Social Security Administration should provide to the Panel, Congress, and the President ongoing interim reporting on the participation of members of the four targeted groups in the Ticket to Work and Self-Sufficiency Program.

**Panel Recommendation 5:****The Panel recommends that Congress approve a technical amendment to require the Social Security Administration to provide to the Panel, Congress, and the President ongoing interim reporting on the participation of members of the four targeted groups in the Ticket to Work and Self-Sufficiency Program. 

**Panel Recommendation 6:** Ongoing data collected and outcomes determined for the Adequacy of Incentives Study should be included in the Social Security Administration’s overall Ticket evaluation research.

**Panel Recommendation 7:****Due to delays in issuing the tickets, the deadline for the Adequacy of Incentives Report to Congress should be extended by the amount of time the ticket rollout has been delayed (13 months). The Panel recommends that Congress approve an amendment to adjust the timeline for the Adequacy of Incentives Report so that it is due 49 months, rather than 36 months, after the enactment date. Implementation of the necessary adjusted payment rates should occur as soon as practical following the release of the Adequacy of Incentives Report but no later than 1 year after its release. Further, the Panel requests preliminary progress reports on the Adequacy of Incentives Study at 36 and 42 months after the date of enactment of the law. 

**Identifying the Four Groups**

The Social Security Administration should not rely on diagnostic data alone in attempting to identify the four groups of interest to Congress. Such efforts will yield less than perfect outcomes. Using standard diagnostic categories that would include but not be limited to developmental disability, mental retardation, psychiatric disability, and multiple disabilities (such as deaf and blind or blind and cerebral palsy) as a major identification tool for the four groups fails to recognize the wide heterogeneity within these categories. The categories that would “most likely” include a significant portion of the population in the four target groups would also include a number of individuals who do not fall within the target groups. Likewise, individuals in the target groups may be found in other diagnostic categories that would not be expected to include them. Moreover, supplementing existing data through surveys is problematic because a high proportion of the population of interest will have disabilities that impede survey participation.

**Panel Recommendation 8:** While there is a need to take advantage of what is known about the diagnostic categories with high proportions of those most likely to fall into one of the four groups relative to work capability, the Social Security Administration, in addition to diagnostic data, should use other information, such as survey information, information on earnings, onset date, and medical improvement assessments. SSA should also go to outside data sources, such as those identified in Expert Roundtable Recommendation 1, to identify members of the four target groups.

**Survey Methodology**

**Panel Recommendation 9:** The Social Security Administration has proposed a survey to supplement its administrative data, improving its ability to target the four groups. If the survey uses a telephone protocol, the protocol should include two other elements. First, there should be face-to-face home interviews if the phone survey fails for a given respondent, except for cases of incorrect phone numbers, empty residences, and so on. Second, if the potential respondent’s disability prevents effective participation in the survey, the interviewer should use as a proxy another individual who knows the respondent well and, preferably, who has communicated on the respondent’s behalf in the past, such as a parent, spouse, sibling, or supervisor in a group home. Members of the Adequacy of Incentives Advisory Team (see Panel Recommendation 2) should review the interview protocol used by the contractor conducting the survey.

**Supplementary Research**

**Panel Recommendation 10:** Recognizing the need for research to supplement what can be studied using Social Security administrative data, the SSA has proposed a program of supplementary research. It has solicited the suggestions of the Panel for possible research topics to supplement the research using SSA data. The Panel recommends the following:

· Review existing research on the four groups or related groups, including relevant behavioral demonstrations and field experiments.

· Conduct or commission new research on these groups using State data or other Federal data.

· Conduct or commission new research related to Employment Networks and other providers.

· Review and consider best practices of successful programs relating to employment of those with significant disabilities.

· Review best practice systems of results-based funding and/or individual milestones payments.

· Conduct or commission a review of supported employment literature, research, and best practices.

Conduct or commission ongoing demonstration projects, modeling, and evaluation of the effectiveness of alternative incentives identified.

**Employment Networks**

**Panel Recommendation 11:** SSA should identify the data elements that Employment Networks (ENs) should collect on whom they serve and do not serve from the four functional groups identified in the statute. The data should include descriptions of why a ticket is accepted or is not accepted on the basis of the EN’s perception of cost, degree of difficulty, or lack of expertise on the part of the EN to provide the necessary services. SSA should use a sampling of ENs rather than requiring all ENs to collect the data. SSA should pay the ENs to offset the administrative burden of management information system design.

**Panel Recommendation 12:****SSA should conduct or commission qualitative research, including interviews and focus groups as appropriate, to assess the adequacy of current incentives to large and small providers, including community rehabilitation programs. Information and recommendations should be collected regarding the amount and timing of payments and other factors that affect potential participation in the Ticket Program and integrated community employment at minimum wage or higher levels for individuals in the four groups.

**Various and Diverse Populations**

**Panel Recommendation 13:** SSA should ensure that the data collected include attention to national demographics such as rural and urban settings, minority representation, and socioeconomic conditions across the country.

**Recommendations From the Experts Roundtable**

Following are the recommendations that emerged from the Experts Roundtable in response to the nine questions posed by the Social Security Administration and the Panel. The Panel strongly urges the Social Security Administration to seriously consider and further develop these recommendations in designing research related to the Adequacy of Incentives and its context within the broader evaluation of the Ticket to Work and Self-Sufficiency Program. 

**Identifying the Four Groups**

**Summary of Comment**

Public comment regarding identifying members of the four subgroups noted the importance of including people with Down syndrome, people with significant disabilities, and people who are deaf and blind. Information was shared about databases on those who are deaf and blind. In addition, comment included the importance of ensuring that the Ticket is available in the transition years from school to adult life. Public comment also emphasized the importance of ensuring that the Ticket incentives are modified in the interest of people whose Social Security benefits may not be eliminated.

**Discussion**

Information from the original design of the study, the Experts Roundtable, and the public testimony show that there will be difficulty in identifying individuals in these four groups in the Federal administrative data available. As a result, it will be necessary to add methods to locate these individuals in order to collect data about use of the Ticket. It is clear that many of the individuals in these four groups are now eligible for, or receiving, services via State developmental disability programs or mental health agencies. Those receiving services and likely to be eligible for the Ticket are likely to be found in day activity programs, day treatment programs, sheltered workshops, supported employment, and other appropriate databases. Because there are important questions about the availability of Federal data, State sources of information will be needed in order to find and understand the participation or nonparticipation of people from the four groups. Sources that can provide information to locate members of the four groups will include State mental retardation and developmental disability agencies; State mental health agencies; providers of sheltered work, day activity, and day treatment programs; providers of supported employment services; and some State vocational rehabilitation agencies. Existing supported employment databases in some States could be very useful. Other databases may be of use, such as available data about people with deafness and blindness and data related to community rehabilitation facilities that serve people in these four groups. Through such sources, it will be possible to identify relatively large samples of people who fit the descriptions of each of the four groups. 

Identifying samples will depend in part on the availability of State-level data regarding people in the four groups. Some States will have databases that make such analysis possible; other States may have a limited ability to provide such data. These data will be useful for several kinds of studies. First, identifying a relatively large sample in each State will allow specific tracking to note the participation or nonparticipation of people in the four groups. In addition, identification of people in the four groups will allow data to be collected from the individuals and their families. In light of the communication issues, it can be expected that it will be necessary, at least in part, to conduct face-to-face interviews and to talk to some family members or significant others regarding participation. The information collected should include the needed disability and demographic information as well as information about knowledge of the Ticket, availability of the Ticket, and access to Employment Networks and providers of service. In addition, it may be possible to gather information about interest in and the feasibility of full-time or near full-time work in order to possibly work enough to eliminate Social Security financial benefits. While such samples will not be random samples from a national perspective, they will provide additional information needed for the Adequacy of Incentives Study.

It must be noted that such State and other databases have serious limits in their completeness and the extent to which they represent in totality the members of the four groups. The four groups, as defined in the Act, have never before been defined in quite this way. As a result, no existing databases can be expected to be complete or representative of the four groups. Because of this, a combination of national administrative data and State and other data should be used to construct the best possible understanding of the groups and the best possible understanding of the incentives to Employment Networks and providers in providing access to the Ticket to members of the four groups. This approach is supported by the Experts Roundtable and by the public input provided to the Panel.

**Roundtable Recommendation 1:** In addition to the administrative data sets SSA has proposed using, use appropriate Federal databases (e.g., Office of Special Education and Rehabilitative Services, Rehabilitation Services Administration, and Department of Labor); State databases (e.g., State developmental disability agencies and State mental health agencies); data from providers of day treatment, day activity programs, sheltered workshops, and supported employment programs (e.g., data from the Employment Intervention Demonstration Program funded by the Center for Mental Health Services); and existing research studies in order to identify and gather information from and about members of the four groups. In addition, the Helen Keller National Center has data on 7,000 persons who are deaf and blind, some of whom are seeking employment.

**Roundtable Recommendation 2:** Oversample potential members of these four groups who can be located in the administrative data from Social Security, Office of Special Education and Rehabilitative Services, Rehabilitation Services Administration, Medicaid, the Department of Labor, and/or other Federal sources.

**Outcomes of Interest**

**Summary of Comment**

Public comment related to outcomes supported the emphasis on basic employment outcomes, including wages and integration in the workforce. However, commenters also indicated that it is important to emphasize that the Ticket should increase choices and control by people with disabilities and their families. Commenters also expressed concern that a phone survey will not be adequate to gather outcome and participation data from the people in the four targeted groups. Commenters also agreed that qualitative methods for collecting data would be needed in addition to quantitative methods. In addition, one commenter emphasized the use of existing databases that include people with significant disabilities. Experts and commenters agreed that an essential outcome is the opportunity for individuals in the four groups to use a ticket to obtain successful employment. There was agreement that successful employment should be defined as real work in integrated settings in the community at minimum or higher wage, based on individual choice and control.

**Discussion**

There are two levels of outcomes of interest: the individual outcomes of interest and the broad program outcomes of interest. The individual outcomes of interest include type of job, wages earned, hours worked, job retention, and the process (and timing) of gaining access to the Ticket through the Employment Networks and providers of services. When members of the four groups do not use or have access to the Ticket, the Social Security Administration will need to know the reasons why individuals are not participating: Is it voluntary or involuntary nonparticipation? It will be important to know if the Ticket Program increases choices of employment services for individuals. It will also be important to know if partial reduction of benefits is possible and helpful to individuals. At a broad program level, it will be important to understand the patterns of participation, outcomes, costs, and incentives that may be adequate to produce desired outcomes. 

**Roundtable Recommendation 3:** Track basic employment outcomes (jobs, wages, hours, choice, etc.) for members of the four groups once identified.

**Roundtable Recommendation 4:** Gather information about access to and use of the Ticket, with particular attention to whether nonparticipation is voluntary or involuntary. 

**Roundtable Recommendation 5:** Conduct or commission qualitative research, including interviews and focus groups, to gather more extensive information about participation, economic and other benefits, and ways to attain improved incentives.

**Roundtable Recommendation 6:** Analyze the data collected related to patterns of outcomes, cost/benefit, participation, and modifications of incentives for members of these four groups compared with others making use of the ticket.

**Maximizing Outcomes**

**Summary of Comment**

Public comment on the array of services emphasized the need for flexibility in how the Ticket Program is used to ensure that members of the four groups have access to the Ticket. Commenters noted that there are a number of best practice programs around the country that can provide information about successful employment of people with significant disabilities. Commenters also noted that the Ticket Program should supplement and not supplant other services received by members of the four groups. In addition, comment included attention to the importance of ensuring that the Ticket Program incentives were adequate for the providers of services. Finally, comments focused on the need to understand results-based funding and milestone payment programs for employment outcomes.

**Discussion**

A third important question concerns the array of services that should be provided to members of these four groups. In considering essential services and supports, it is important to recognize that a large number of people who typically would fall into one of the four groups have become successfully employed through an array of services and supports. It is critical that the study include an evaluation of the array of services and supports that led to successful employment. The experiences of the supported employment initiative, the Rehabilitation Services Administration Choice projects, and the milestone payment systems in vocational rehabilitation will be especially relevant in this regard. In addition, the combination of services (vocational rehabilitation, developmental disabilities programs, mental health programs, Medicaid, etc.) received by some members of these four groups will be important to understand. For people in these four groups, there are other macro and micro factors that relate to the opportunity to work. Such factors include benefits planning, assistance and outreach projects, availability (and payment methodology, for example, fee-for-service vs. results-based funding) of vocational rehabilitation funds, availability and structure of long-term employment support funds (via mental retardation/developmental disability and mental health agencies), use of Medicaid funds, Plan for Achieving Self-Support and Impairment-Related Work Expense use, housing allowances and funding, and the perception of disincentives to employment. Factors such as these already complicate work decisions for people in these four groups and have an impact on their unemployment or underemployment. In addition, issues of continued health care coverage and States’ use of the Medicaid Buy-In process will have an impact on employment. Analysis of the Adequacy of Incentives will depend in part on analysis of this complex context and of possible interaction effects. There are also related questions about combining funding sources in the interest of employing these individuals. Another area for analysis is the extent to which those eligible for tickets are sufficiently motivated to work.

**Roundtable Recommendation 7:** The Social Security Administration should ensure that studies of these four groups include attention to the issue of the “mix” of funding, government programs, and other incentives and disincentives that affect the employment of people with disabilities. Studies should include analysis of how various funding streams can be used together to enhance incentives to Employment Networks and providers and result in employment for people in the four groups.

**Successful Efforts to Employ People With Significant Disabilities in Community Employment at Minimum Wage Levels or Higher**

**Summary of Comment**

The public comment about experts and information resources to consider focused on initiatives in the past 10 to 15 years that have emphasized employment for people with severe disabilities. These initiatives include supported employment, the Rehabilitation Services Administration Choice projects, and other demonstration projects, including those for persons with mental illness as well as persons who are deaf and blind.

**Discussion**

Public comment and the discussion of the Experts Roundtable emphasized that members of the four groups have benefited from employment initiatives in the recent past. Evidence of these successful strategies is described in some detail in the supported employment literature, in reports regarding projects to create greater choice and control in the vocational rehabilitation process, and in a number of demonstration projects that have been created in the past 15 years. Evidence of such projects is summarized in a number of books and published research reports noted in the bibliography. While these initiatives have not solved the widespread unemployment or underemployment of people in the four groups, there is ample evidence that these people can work if they are provided with creative and individualized supports. Study of these successes can help inform the process of discovering how to modify incentives related to the Ticket.

**Roundtable Recommendation 8:****The Social Security Administration should study the successes of supported employment programs, Rehabilitation Services Administration Choice projects, and other demonstration projects of the past decade for members of these four groups.

**Other Methods to Identify Members of the Four Groups**

**Summary of Comment**

Public comment supported the concern about administrative data and the planned phone surveys as a way to gather information from members of the target groups. Comment included an emphasis on the need to consult other databases that may be from States, regions, or national samples of persons likely to be members of the four groups.

**Discussion**

There is concern that administrative data will not be adequate to locate members of the four groups. As a result, commenters and the Experts Roundtable emphasized the need to use other databases to find people in the four groups in order to gather information about their participation and about ways to modify the incentives of the Ticket. Owing to the heterogeneity of the population with disabilities, large samples and information specifically needed to analyze key subgroups would be especially useful. 

**Roundtable Recommendation 9:****The Social Security Administration should consult other databases to identify members of the four groups. Such databases should include other existing national samples as well as State databases for persons with developmental disabilities, persons with mental illness, persons with deafness and blindness, persons with brain injuries, and persons with other disabilities.

**Individualized Milestone Payments**

**Summary of Comment**

Public comment and the Experts Roundtable cited the recent experiences in Massachusetts, Oklahoma, and other States with creating milestone payments in the interest of employment for people with disabilities and including members of the four groups.

**Discussion**

Results-based funding and milestone payment systems are emerging in at least a dozen States across the country (e.g., Massachusetts and Oklahoma) in the interest of increasing employment for people with significant disabilities, some of whom are members of the four groups. These experiences can provide information about effective means of funding providers of services to create employment outcomes. They can also provide a framework for creating demonstration projects that make use of emerging payment mechanisms in combination with best practices for employment programs.

**Roundtable Recommendation 10:****The Social Security Administration should study the current and recent experiences of States that have begun using results-based funding or some form of payment for individualized milestones toward employment.

**Roundtable Recommendation 11:****The Social Security Administration should conduct or commission cost/benefit analyses and create demonstration projects on modifying the incentives to Employment Networks and providers of services, showing how members of the four groups can benefit from the Ticket and from employment that partially reduces benefits.

**Various and Diverse Populations**

**Summary of Comment**

Input related to diversity issues included the need for attention to the unique differences of minorities related to gaining access to employment. Concern was also expressed about regional differences nationwide and about Native American people.

**Discussion**

Because of the diversity in the country, it will be important to ensure that the study design is sensitive to a range of diversity issues. National demographics should be considered in the design and implementation of the study, with particular attention to rural and urban settings, minority representation, and emerging economic issues.

**Roundtable Recommendation 12:** The Social Security Administration should ensure that the data collected include attention to national demographics such as rural and urban settings, minority representation, and socioeconomic conditions across the country.

**Including Consumers, Family Members, and/or Their Representatives in the Study Design and Implementation**

**Summary of Comment**

Input related to participation in the study design process noted the need for people with disabilities and families to be included. Others commented that research methodology used should be accessible and not limit participation.

**Discussion**

Because participation or nonparticipation in the Ticket Program will profoundly affect people with disabilities and their families, as well as the overall success of the program, it is important that they clearly be a part of the process for design, implementation, and ongoing tracking of the Adequacy of Incentives evaluation.

**Roundtable Recommendation 13:****Given the impact of this and other studies the Office of Policy conducts on the lives of people with disabilities and their families, the Office of Policy should develop a policy and practice for incorporating the input and experiences of consumers, their families, and/or their representatives as it develops new research designs, including the Adequacy of Incentives Study. 

**Including Employment Networks in the Study Design and Implementation**

**Summary of Comment**

Input regarding the role of Employment Networks and providers emphasized the need for the incentives to be clearly adequate from a financial standpoint for both large and small providers. Commenters noted that the Ticket payment schedule might be too far in the future for providers to believe the incentive will be adequate. There was also some concern about the possibility that Employment Networks would experience an excessive reporting burden if they were not told in advance—before being contracted—about the extent of information needed. It was further suggested that Social Security Administration consider random sampling of Employment Networks to obtain this information and consider providing a financial incentive for participation in gathering this information. 

**Discussion**

As tickets are released and Employment Networks begin service to beneficiaries, it is critical to ensure that a stream of information and data is steadily collected and analyzed relative to how Employment Networks serve beneficiaries in these four categories. To do this effectively, the Social Security Administration should identify up front what specific data need to be collected, to enable ENs to incorporate the data collection into their database design. To further facilitate accurate and timely collection, SSA should consider asking for the data from a representative sample of ENs rather than from all ENs and should pay a reasonable fee to participating ENs to cover the cost of the administrative burden. In addition to collecting data on those served by an EN, data should be collected on those whose tickets the EN did not accept. It is expected that some ENs will reject tickets on the basis of the perception, real or imagined, that people in the four targeted groups will ultimately cost more to serve than is available in the outcome/milestone payment program. Surveys and focus groups could be used to garner recommendations from ENs as to what they believe sufficient incentives to be. Incentives could be measured in terms of time, money, and difficulty (e.g., availability of certain services in rural areas, coordination with the One-Stop Career Centers for job skills training funds, cooperation with the State vocational rehabilitation agency). Social Security should compare data among ENs to determine whether or not there is consistency among and between ENs in terms of whether they accept tickets, costs they claim to incur for various services, and perceived difficulty in serving consumers in the four targeted groups.

**Roundtable Recommendation 14:** The Social Security Administration should gather data related to the costs to Employment Networks and providers of services and the timeframes for providing access to the Ticket and employment for members of these four groups.

**Roundtable Recommendation 15:** The Social Security Administration should study the relationship between perceived disincentives and actual incentives for Employment Networks and providers of services, including incentives for small providers of services.

Summary

This Advice Report provides a framework for recommendations related to the design of the Adequacy of Incentives Study. This guidance is intended to enhance and supplement, not supplant, the overall design proposed by the Social Security Administration in order to understand the impact of the Ticket Program for members of the four noted groups and to determine how incentives can be modified to promote access to employment services and supports for all beneficiaries of Social Security Administration disability benefit programs. 

**APPENDIX 1:**

*TICKET TO WORK AND WORK INCENTIVES ADVISORY PANEL*

Sarah Wiggins Mitchell, R.N., M.S.W., J.D., ChairPresident and Executive Director New Jersey Protection and Advocacy, Inc. 

Richard V. Burkhauser, Ph.D. Professor of Policy Analysis Chair, Department of Policy Analysis and Management Cornell University Ithaca, NY 

Kristen E. Flaten, M. Div. Employment Consultant Lifetrack Resources St. Paul, MN 

Thomas P. Golden, M.S., C.R.C. Extension Faculty Program on Employment and Disability School of Industrial and Labor Relations Cornell University Ithaca, NY 

Frances Gracechild Executive Director Resources for Independent Living, Inc. Sacramento, CA 

Christine M. Griffin, J.D. Executive Director Disability Law Center Boston, MA 

Larry D. Henderson Executive Director Independent Resources, Inc. Wilmington, DE 

Jerome Kleckley, M.S.W., C.S.W. Director Hospital Services Eastern Paralyzed Veterans Association Jackson Heights, NY 

Stephanie Smith Lee Governmental Affairs National Down Syndrome Society****Oakton, VA 

Bryon R. MacDonald Public Policy Advocate World Institute on Disability Oakland, CA 

Stephen L. Start President and CEO S.L. Start Associates Spokane, WA 

Susan Webb  President  Webb Transitions, Inc.  Phoenix, AZ

**Appendix 2:**

*DESIGN AND EVALUATION COMMITTEE*

Thomas P. Golden, M.S., C.R.C., Chair Extension Faculty Program on Employment and Disability School of Industrial and Labor Relations Cornell University Ithaca, NY 

Richard V. Burkhauser, Ph.D. Professor of Policy Analysis Chair, Department of Policy Analysis and Management Cornell University Ithaca, NY 

Kristen E. Flaten, M. Div. Employment Consultant Lifetrack Resources St. Paul, MN 

Stephanie Smith Lee Governmental Affairs National Down Syndrome Society Oakton, VA 

Bryon R. MacDonald Public Policy Advocate World Institute on Disability Oakland, CA 

Susan Webb  President  Webb Transitions, Inc.  Phoenix, AZ

**Appendix 3:**

*LIST OF EXPERTS*

*EXPERTS ROUNDTABLE, JULY 27, 2001*

Peter Blanck Professor of Law, Director Law, Health Policy and Disability Center University of Iowa Iowa City, IA  

Crystal R. Blyler, Ph.D. Social Science Analyst Center for Mental Health Services Rockville, MD  

Jane Burke, M.S. Research Specialist and Project Manager University of Illinois at Chicago Mental Health Services Research Program Chicago, IL  

Claire T. GhiLoni Director Statewide Employment Services Massachusetts Rehabilitation Commission Boston, MA  

Richard L. Horne, Ed.D. Senior Policy Advisor Presidential Task Force on Employment of Adults With Disabilities Washington, DC 

William E. Kiernan, Ph.D. Director Institute for Community Inclusion Boston, MA  

John Kregel Associate Director – VCU RRTC****Virginia Commonwealth University Richmond, VA  

David Mank Director Indiana Institute on Disability Bloomington, IN 

Constance Pledger, Ph.D. Project Officer NIDRR U.S. Department of Education Washington, DC  

Ronald E. Rucker President and CEO VIA of the Lehigh Valley, Inc. Allentown, PA  

Priscilla Sanderson Director American Indian Research and Training Center****Arizona University Affiliated Program Institute for Human Development  Northern Arizona University Flagstaff, AZ ****

Robert Silverstein Director Center for the Study and Advancement of Disability Policy Washington, DC  

Sue SwensonCommissionerAdministration on Developmental DisabilitiesU.S. Department of Health and Human Services Washington, DC 

Appendix 4:

*PUBLIC COMMENTERS, JULY 27, 2001*

Dr. William Halleran U.S. Department of Education Washington, DC 

Jim Larson President and CEO Morningside Olympia, WA 

Susan Goodman Director of Governmental Relations National Down Syndrome Congress****Oakton, VA 

Jan Hoyme Employment Link Boulder, CO 

Jamie Ruppman Director of Governmental Affairs TASH****Washington, DC 

Joseph McNulty Director Helen Keller National Center New York, NY

**Appendix 5:**

*INDIVIDUALS PROVIDING PUBLIC COMMENT IN WRITING*

*FOLLOWING JULY 27, 2001*

Barbara Carroll The Arc of Tuscaloosa County

Brenda D. Carson United Cerebral Palsy of Greater Birmingham, Inc.

Sue GunnRegion VII Community Rehabilitation Regional Continuing Education Program

David L. HorneOklahoma Association for Persons in Supported Employment

Diane Kaatz Wildwood Programs

Ed Kaler Mid-Coast Mental Health Center

Robert A. Lawhead Boulder, CO

Debra Martin Luecking University of Tennessee

Brenda McComb Alabama Department of Mental Health and Mental Retardation

Celane McWhorter Association for Persons in Supported Employment 

Paul Mendez National Association of Workforce Development Professionals

Leslie K. Paull Ohio Developmental Disabilities Planning Council

Susan Rinne Options for Better Living, Inc.

Carol Rydell Advocate

Steve Savage Indiana Association for Persons in Supported Employment 

Yvonne Swanson Human Technologies Corporation

Peggy Todd University of Missouri at Columbia

Appendix 6:

*STAFF, JULY 27, 2001*

**Social Security Administration Staff**

Steve Fear

Minh Huynh

Howard Iams

Deborah Morrison

Paul O’Leary

**Advisory Panel Staff**

Marie Strahan, Executive Director 

Gordon Richmond

Theda Zawaiza

Ilene Zeitzer

**Appendix 7:**

*PRESENTERS, JULY 27, 2001*

Jane Burke, M.S.Research Specialist and Project ManagerUniversity of Illinois at Chicago Mental Health Services Research ProgramChicago, IL

Gina LivermoreResearch AssociateCornell Center for Policy ResearchWashington, DC 

Paul O’LearyProject DirectorOffice of PolicySocial Security AdministrationWashington, DC

Priscilla SandersonDirectorAmerican Indian Rehabilitation Research and Training CenterArizona University Affiliated ProgramInstitute for Human Development Northern Arizona UniversityFlagstaff, AZ 

**APPENDIX 8:**

*References and Bibliography*

Albin, J. M., Rhodes, L., and Mank, D. (1994). Changeover to community employment: The problem of realigning organizational culture, resources, and community roles. *Journal of the Association for Persons with Severe Handicaps*, *19*(2), 105–115.

Bellamy, G. T., Horner, R. H., and Inman, D. P. (1979). *Vocational habilitation of severely retarded adults: A direct service technology*. Austin, TX: PRO-ED.

Bellamy, G. T., Rhodes, L. E., Mank, D. M., and Albin, J. M. (1988). *Supported employment: A community implementation guide*. Baltimore: Paul H. Brookes Publishing Co.

Berkell, D. E. (1987). Vocational assessment of students with severe handicaps: A review of the literature. *Career Development for Exceptional Individuals*, *10*(2), 61–75.

Beyer, S. (1999). Relationships between agency organization, cost, and worker outcomes—the S.E.P.O.D. project. Paper presented at the European Union of Supported Employment, Rome.

Braddock, D. (2001). Public financial support for disability programs at the close of the 20th century. A working paper. Boulder, CO: Coleman Institute for Cognitive Disabilities, University of Colorado.

Braddock, D., Hemp, R., Parish, S., and Rizzolo, M. C. (2000).*The state of the states in developmental disabilities: 2000 study summary*. Chicago: University of Illinois at Chicago, Department of Disability and Human Development.

Buckley, J. T., and Bellamy, G. T. (1985). National survey of day and vocational programs for adults with severe disabilities: A 1984 profile. In P. Ferguson (Ed.), *Issues in transition research: Economic and social outcomes* (pp. 1–12). Eugene, OR: Specialized Training Program, University of Oregon.

Butterworth, J., and Gilmore, D. (2000). Are we there yet? Trends in employment opportunities and supports. *TASH Newsletter*, *26*(6/7), 5–7.

Butterworth, J., Gilmore, D., and Kiernan, W. E. (2000). *Trends and issues in sheltered employment services*. Boston: Children’s Hospital, Institute for Community Inclusion.

Butterworth, J., Gilmore, D., Kiernan, W. E., and Schalock, R. (1999). *State trends in employment services for people with developmental disabilities: Multiyear comparisons based on state MR/DD agency and vocational rehabilitation (RSA) data*. Boston: Children’s Hospital, Institute for Community Inclusion.

Callahan, M. (2000, December). Choice and self-determination in employment.*Common Sense,**8*, 1–4.

Callahan, M. J., and Garner, J. B. (1997). *Keys to the workplace: Skills and supports for people with disabilities*. Baltimore: Paul H. Brookes Publishing Co.

Callahan, M., and Mank, D. (1997). Choice and control of employment for people with disabilities. Paper prepared for The Robert Wood Johnson Foundation Self-Determination Initiative, Princeton, NJ.

Campbell, D. T., and Stanley, J. C. (1963)*. Experimental and quasi-experimental designs for research*. Boston: Houghton Mifflin Company.

Carr, E. G., Horner, R. H., Turnbull, A. P., Marquis, J. G., McLaughlin, D. M., McAtee, M. L., Smith, C. E., Ryan, K. A., Ruef, M. B., and Doolabh, A. (1999). In D. Braddock (Ed.),*Positive behavior support for people with developmental disabilities: A research synthesis*. Washington, DC: American Association on Mental Retardation.

Church, G., and Glennen, S. (1992). *The handbook of assistive technology*. San Diego, CA: Singular Publishing Group, Inc.

Everson, J. M. (1995). *Supporting young adults who are deaf-blind in their communities: A transition planning guide for service providers, families, and friends*. Baltimore: Paul H. Brookes Publishing Co.

Gaylord-Ross, R. (1986). The role of assessment in transitional, supported employment. *Career Development for Exceptional Individuals*, *9*(2), 129–134.

Gilmore, D., Schuster, J., Timmons, J., and Butterworth, J. (2000). An analysis of trends for people with mental retardation, cerebral palsy, and epilepsy receiving services from state vocational rehabilitation agencies: Ten years of progress. *Rehabilitation Counseling Bulletin*, *4*(1), 30–38.

Gold, M. (1973). Research on the vocational rehabilitation of the retarded: The present, the future. In N. Ellis (Ed.), *International review of research in mental retardation* (Vol. 6, pp. 97–147). New York: Academic Press.

Golden, T. P., O’Mara, S., Ferrell, C., and Sheldon, J. (2001). *Benefits planning, assistance and outreach: Supporting the career development and employment of people with disabilities.*Publication Number 63-003. Washington, DC: U.S. Government Printing Office, Social Security Administration. 

Golden, T. P., O’Mara, S., and Czechowicz, J. (1998). Plans for achieving self-support: An improved tool for self-directed vocational rehabilitation. *Journal of Applied Rehabilitation Counseling, 29*, 4.

Griffin, C. (1998). *Rural facts: Rural supported employment*. Missoula, MT: Research and Training Center on Rural Rehabilitation Services, Montana University Affiliated Rural Institute on Disabilities, University of Montana. Retrieved January 16, 2001, from the World Wide Web:  GOTOBUTTON BM_1_ http://ruralinstitute.umt.edu/rtcrural/Pubs/allRTCPubs.htm.

Guba, E. G. (1978). *Toward a methodology of naturalistic inquiry in educational research.*CSE Monograph Series in Evaluation. 8. Los Angeles: Center for the Study of Evaluation, UCLA Graduate School of Education.

Human Services Research Institute (HSRI). (1999). Consumer Survey. Cambridge, MA: HSRI.

Huynh, M., and O’Leary, P. K. (2001). Design of the Study of the Adequacy of Incentives for the Ticket to Work Program. Working Paper 003. Washington, DC: Office of Policy, Social Security Administration.

Hughes, C., and Carter, E. W. (2000).*The transition handbook. Strategies high school teachers use that work!* Baltimore: Paul H. Brookes Publishing Co.

Izzo, M. (2000). National longitudinal transition study. Presentation at World Congress on Disability, Atlanta, GA.

Izzo, M. V., Cartledge, G., Miller, L., Growick, B., and Rutkowski, S. (2000). Increasing employment earnings: Extended transition services that make a difference. *Career Development for Exceptional Individuals*, *23*(2), 139–156.

Jenaro, C., Mank, D., Bottomley, J., Doose, S., and Tuckerman, P. (in press). Supported employment in the international context: An analysis of processes and outcomes. *Journal of Vocational Rehabilitation*.

Kiernan, W. E., and Schalock, R. L. (Eds.). (1997). *Integrated employment: Current status and future directions*. Washington, DC: American Association on Mental Retardation.

Koegel, L. K., Koegel, R. L., and Dunlap, G. (1996). *Positive behavioral support: Including people with difficult behavior in the community*. Baltimore: Paul H. Brookes Publishing Co.

Koehler, F., and Ellis, J. (Eds.). (1990). *Project WIN, Work Incentives Network, training manual*. Richmond, VA: Association for Persons in Supported Employment. 

Kregel, J. (2001). *Potential strategies for “Operationalizing” the target populations defined in Section 101 (h) (5) (c) of P. L. 106-70, The Ticket to Work and Work Incentives Improvement Act.* Richmond, VA: Virginia Commonwealth University.

Mank, D. M. (1994). The underachievement of supported employment: A call for reinvestment. *Journal of Disability Policy Studies*, *5*(2), 1–24.

Mank, D. M. (1996). Evolving roles for employers and support personnel in the employment of people with severe disabilities. *Journal of Vocational Rehabilitation*, *6*(1), 83–88.

Mank, D. (1997). Systems change strategies for integrated employment: A blueprint for the future. In W. E. Kiernan and R. Schalock (Eds.), *Integrated employment: Current status and future directions* (pp. 107–120). Washington, DC: American Association on Mental Retardation.

Mank, D., Buckley, J., Cioffi, A., and Dean, J. (1996). Do social systems really change? Retrospective interviews with state supported employment systems change project directors. *Journal of Autism and Developmental Disabilities*, *11*(4), 243–250.

Mank, D., Cioffi, A., and Yovanoff, P. (1997). Patterns of support for employees with severe disabilities. *Mental Retardation*, *35*(6), 433–447.

Mank, D., Cioffi, A., and Yovanoff, P. (1998). Employment outcomes for people with severe disabilities: Opportunities for improvement. *Mental Retardation*, *36*(3), 205–216.

Mank, D., Cioffi, A., and Yovanoff, P. (1999). Impact of coworker involvement with supported employees on wage and integration outcomes. *Mental Retardation*, *37*(5), 383–394.

Mank, D., O’Neill, C. T., and Jensen, R. (1998). Quality in supported employment: A new demonstration of the capabilities of people with severe disabilities. *Journal of Vocational Rehabilitation*, *11*(1), 83–95.

McDowell, J., and Ferrell, C. (Eds.). (2000). *Project DOOR’s guide to using social security work incentives*. Bloomington, IN: Indiana University, Indiana Institute on Disability and Community.

McGaughey, M., and Mank, D. (2001). Empirical evidence of systems change in supported employment. *Journal of Disability Policy Studies*, *11*(4), 210–222.

McGaughey, M., and Mank, D. (2001). State policy innovation and systems change: The context for the supported employment initiative. *Journal of Disability Policy Studies*, *11*(4), 202–209.

McMillan, J. H., and Schumacher, S. (1984). *Research in education: A conceptual introduction*. Boston: Little, Brown and Company.

Menchetti, B. M., and Rusch, F. R. (1988). Vocational evaluation and eligibility for rehabilitation services. In P. Wehman and M. S. Moon (Eds.), *Vocational rehabilitation and supported employment* (pp. 79–90). Baltimore: Paul H. Brookes Publishing Co.

Moon, M. S., Inge, K. J., Wehman, P., Brooke, V., and Barcus, J. M. (1990). *Helping persons with severe mental retardation get and keep employment: Supported employment issues and strategies*. Baltimore: Paul H. Brookes Publishing Co.

Murphy, S. T., and Rogan, P. M. (1995). *Closing the shop: Conversion from sheltered to integrated work*. Baltimore: Paul H. Brookes Publishing Co.

Nisbet, J. (1992). *Natural supports in school, at work, and in the community for people with severe disabilities*. Baltimore: Paul H. Brookes Publishing Co.

Novak, J. (2000). *Changing supported employment systems: Findings from a national survey of state vocational rehabilitation agencies. Preliminary report*. Bloomington, IN: Indiana Institute on Disability and Community, Indiana University.

Novak, J., Mank, D., Revell, G., and O’Brien, D. (1999). Paying for success: Results-based approaches to funding supported employment. In G. Revell, K. Inge, D. Mank, and P. Wehman (Eds.), *The impact of supported employment for people with significant disabilities: Preliminary findings from the National Supported Employment Consortium* (pp. 25–42). Richmond, VA: Virginia Commonwealth University.

Olson, D., Cioffi, A., Mank, D., and Yovanoff, P. (2001). Employers’ perceptions of employees with mental retardation.*Journal of Vocational Rehabilitation, 16*(2), 125–133. 

Presidential Task Force on Employment of Adults with Disabilities. (2000, December). *Recharting the course: Turning points.* (Third report.) Washington, DC: Author.

Revell, G., Inge, K. J., Mank, D., and Wehman, P. (Eds.). (1999). *The impact of supported employment for people with significant disabilities: Preliminary findings from the National Supported Employment Consortium*. Richmond, VA: Virginia Commonwealth University.

Rogan, P., Banks, B., and Howard, M. (2000). Workplace supports in practice: As little as possible, as much as necessary. *Focus on Autism and Other Developmental Disabilities*, *15*(1), 2–11.

Sands, D. J., and Wehmeyer, M. L. (1996). *Self-determination across the life span: Independence and choice for people with disabilities*. Baltimore: Paul H. Brookes Publishing Co.

Schalock, R. L., and Braddock, D. L. (Eds.). (1999). *Adaptive behavior and its measurement: Implications for the field of mental retardation*. Washington, DC: American Association on Mental Retardation.

Sheldon, J., and Trach, J. S. (1998). SSDI and SSI work incentives with recommendations for policy change.*Journal of Applied Rehabilitation Counseling, 29*, 4.

Sowers, J., and Powers, L. (1991). *Vocational preparation and employment of students with physical and multiple disabilities*. Baltimore: Paul H. Brookes Publishing Co.

Stiles, K. (1999). *Beyond borders: Global supported employment and people with disabilities*. St. Augustine, FL: Training Resource Network, Inc.

Taylor, H. (2000)*. Harris Poll #59, Conflicting trends in employment of people with disabilities—1986–2000*. Retrieved January 16, 2001, from the World Wide Web: http://www.harrisinteractive.com/harris_poll/index.asp?PID=121.

U.S. Department of Health and Human Services. (2001). *Olmstead update no. 5* (SMDL #01-007). Baltimore, MD: Author.

Unger, D. D. (1999). Workplace supports: A view from employers who have hired supported employees. *Focus on Autism and Other Developmental Disabilities*, *14*(3), 167–179.

Wehman, P. (2001). *Life beyond the classroom. Transition strategies for young people with disabilities.*3rd ed. Baltimore: Paul H. Brookes Publishing Co.

Wehman, P. (Ed.). (2001). *Supported employment in business: Expanding the capacity of workers with disabilities*. St. Augustine, FL: Training Resource Network, Inc.

Wehman, P., and Bricout, J. (1999). Supported employment: Critical issues and new directions. In G. Revell, K. Inge, D. Mank, and P. Wehman (Eds.), *The impact of supported employment for people with significant disabilities: Preliminary findings from the National Supported Employment Consortium* (pp. 1–24). Richmond, VA: Virginia Commonwealth University.

Wehman, P., and Moon, M. S. (1988). *Vocational rehabilitation and supported employment*. Baltimore: Paul H. Brookes Publishing Co.

Weiss, C. H. (1972). Evaluation Research: Methods of assessing program effectiveness. Englewood Cliffs, NJ: Prentice Hall, Inc. 

1

1

PAGE