**Research Brief**

**Independent Evaluation: Interim Report on Accomplishments and Issues**
[^1]

The Office of Disability Employment Policy (ODEP) of the US Department
of Labor (DOL) has funded pilot demonstration programs across the United
States to build the capacity and change the workforce development system
to better serve adults and youths with disabilities. In order to
evaluate the effectiveness of these demonstration programs, ODEP
contracted with the private research firm Westat to conduct an
independent evaluation of these programs. This document summarizes the
findings, conclusions, and recommendations to date of Westat's
evaluation of ODEP's demonstration programs.

This Research Brief summarizes Westat's synthesis report *Interim Report
on ODEP Demonstration Programs* available in [PDF
format](http://www.dol.gov/odep/research/DemoPrograms.pdf) or as a [Word
document](http://www.dol.gov/odep/research/DemoPrograms.doc)[.]{.underline}

The independent evaluation has three objectives:

1\. To provide ODEP with reliable and valid indicators of program
effectiveness.

2\. To determine the extent to which each program priority area is
effective in building workforce development system capacity; and

3\. To document local, regional, and/or state systems change that
supports program effectiveness.

Westat collected both quantitative and qualitative data to generate
their conclusions and recommendations.

-   **Quantitative Data** -- Westat obtained data from a sample of the
    customers who had obtained competitive employment as a result of
    receiving services from the adult demonstration projects.

-   **Qualitative Data** - Westat conducted site visits at all ODEP
    demonstration locations and interviewed the directors of these
    projects, employees and partners of the projects, customers and
    their families, and community representatives. To capture changes
    over time and collect longitudinal data, the Westat researchers will
    return to locations for multiple visits.

    **Overview and Definition of Terms**

    (For definitions of the terms in the paragraph below, click on the
    italicized term.)

ODEP funds three types of demonstration programs that are expected to
impact [*systems change*](../SystemsChange.doc) within agencies,
organizations, and processes under the [*Workforce Investment Act
(WIA)*:](../WIA.doc) 1) [*Youth programs*](../YouthPrograms.doc) to
provide services to help youth with disabilities reach their educational
and occupational goals; 2) Programs for adults with disabilities to
utilize the concepts of [*Customized Employment*](../CustomEmploy.doc);
and 3) [*Technical Assistance*](../TechAssistance.doc) to the youth and
adult demonstration programs as well as other elements of the workforce
investment system.

**Westat's Findings and Conclusions**

**Employment Outcomes**

Westat has twice collected information on employment placements among
randomly selected program participants of the demonstration projects. In
the first round, information was obtained on 345 customers from 25 ODEP
Customized Employment demonstration projects who had been placed in
competitive employment. It was found that the customized employment
process can be effective in helping some of the most hard-to-serve
individuals with disabilities find better employment and allow them to
begin to move off income support programs by obtaining better
employment.

Approximately 43 percent of those who self-disclosed their disability
said that they had psychiatric or emotional disabilities, and 37 percent
had never worked or had only worked in a non-competitive employment
environment. The majority of these respondents found high quality
employment after participating in the ODEP demonstration programs: 95
percent were now earning the minimum wage or higher in jobs they
obtained through the program. More than one-third of these respondents
were earning more than \$8.15 an hour while more than 17 percent were
receiving benefits (such as health insurance or paid vacations) from
their employer. More than half (54 percent) of program customers who had
been placed in competitive employment obtained jobs with the potential
for career advancement.

Compared to their status at the beginning of their involvement with the
project, the percentage of customers of the ODEP demonstration programs
who were on government benefits (such as Food Stamps, Social Security
Income, Social Security Disability Income, and Temporary Aid to Needy
Families) had declined in every category except subsidized housing
(which increased as customers of the Chronic Homelessness were placed in
housing as part of their participation in that program).

The second round of data collection on competitive employment placements
yielded similar findings. Westat collected data on 536 randomly selected
individuals who were program customers in 30 ODEP demonstration
projects. Approximately 42 percent self-disclosed a psychiatric or
emotional disability, and 1 in 5 reported a cognitive disability. Every
program customer in this sample received at least one type of customized
employment service, and each customer received an average of 5.6
customized employment services.

Westat found that 99 percent of all the respondents obtaining employment
earned above minimum wage. One-third now have full-time jobs (more than
35 hours a week), and 27 percent have jobs in which their employer
offers at least one fringe benefit. It was also reported that 63 percent
of the sampled program customers have obtained employment with future
career potential and that 14 percent went off at least one public
benefit after receiving customized employment services.

**Findings from the Westat Site Visits**

Based on extensive interviews conducted during site visits to all of the
ODEP projects and TA centers, Westat reported the following:

-   **Customization of Services --** Historically, most One-Stop Career
    Centers have been oriented towards a self-service model that offers
    a "one size fits all" approach to customers. This resulted in people
    with disabilities being referred elsewhere because the One-Stop
    Career Centers were physically or programmatically inaccessible or
    the staff believed that other programs such as Vocational
    Rehabilitation could better serve people with disabilities. The ODEP
    projects have introduced the person-centered customized approach as
    an alternative to this model. This customized approach has been
    implemented in all demonstration projects. The goal is to improve
    and increase the use of individualized services for people with
    disabilities and test them in the local environment before further
    dissemination and replication to other One Stop Career Centers
    across the country.

-   **WIA Performance Measures --** The ODEP projects have almost
    universally reported that the current indicators of performance for
    workforce investment activities collected under the WIA are a
    disincentive for serving customers with disabilities. Not only are
    One Stop Career Centers reluctant to enroll participants with
    disabilities because of these measures, but some members of the WIA
    system are reluctant to participate in these projects because they
    fear that they will be penalized as their staff members will be
    required to take on time-consuming responsibilities that will
    negatively affect their WIA defined performance metrics.

-   **Characteristics of Program Participants** - According to the
    quarterly reports filed by the demonstration projects, 4,090 youth
    and 4,865 adults have been served by the projects (as of June 2005).
    Among the participants in the youth programs, boys outnumbered the
    girls, more whites than blacks or Hispanics were in the programs,
    and when it was self-disclosed, the most frequent disability type
    was a learning disability. Among the participants in the adult
    programs, more participants were male, white, had completed some
    type of high school education (diploma, GED, or certificate of
    completion) and the most frequent self-reported disability was
    psychiatric or emotional.

-   **Resistance to Full-Time Employment** -- Virtually every project
    location could report customers that did not want to obtain
    full-time employment -- and in some cases turned down an offer of
    full-time employment - because they were afraid of losing their
    government-supplied benefits (especially their financial and medical
    benefits) if they were hired full-time. Several sites have
    instituted benefits planning mechanisms and training for One Stop
    Career Center staff and project partners on benefits, work incentive
    programs, and other financial issues.

-   **Strategic Planning** -- Although the programs were required to
    engage in strategic planning activities, many were still using a
    "plan as you go" approach to planning strategies. Conversely, a
    number had reinforced their understanding of local needs by
    gathering additional information from key stakeholders and bringing
    stakeholders on as part of advisory committees. (However, some
    people with disabilities had difficulty serving on advisory
    committees primarily because of timing or lack of transportation.)

-   **Physical and Programmatic Accessibility** -- Although ODEP's
    project funds could not be used for physical modifications, the
    projects served as an impetus for the sites to modify buildings or
    purchase equipment for physical and communication accessibility. To
    increase the accessibility of their programs, some One Stop Career
    Centers have modified their orientation sessions to make them more
    "disability-friendly." Disability awareness training has also been
    offered to their staff. Some have connected with the local
    Disability Service Navigator run by the Social Security
    Administration and the Department of Labor's Employment and Training
    Administration. Westat postulated that accessibility changes were
    easier to make if the project worked with the One Stop Career Center
    than if the project were implemented by an organization outside the
    purview of the One Stop Career Center.

-   **Building Capacity Through Training --** The demonstration programs
    provided training (including formal training and technical
    assistance) on the following topics for their staff, partners, and
    collaborators: disability awareness, working with people with
    disabilities, availability of resources and services for people with
    disabilities, the available government programs, and advising
    customers on how to obtain financial work incentives. However, as
    Westat noted, the quality of this training varied from location to
    location -- ranging from basic and informational to more thorough
    and problem-solving. Westat also noted that because of staff
    turnover at the One Stop Career Centers, it was necessary to repeat
    staff training to maintain sustainability.

-   **Training on Financial Issues --** Adults and youth (and their
    families) are also receiving training on financial issues (such as
    the impact of employment on benefits, work incentive programs, and
    individual budgets).

-   **Coordination of Services and Resources --** In all of the
    demonstration programs, coordination of services for people with
    disabilities (such as job accommodations, coaches, personal
    assistants, funds for training and education, and transportation)
    has begun with extensive collaboration among partners and
    collaborators. In many locations this coordination has transitioned
    to leveraging of resources among partners and collaborators, and in
    others it has resulted in the establishment of formal arrangements
    among the partners so that these leveraged resources can be
    sustained after the funding to the program has ended. However, the
    range and types of partners varies among programs as do the success
    of these collaborations.

-   **Promising Practices, Program Evaluation, and Dissemination --** A
    number of promising practices have been documented and disseminated
    by the projects, by the Technical Assistance centers, and by ODEP.
    One example of the programs' effectiveness is the way that some
    customers labeled as "unemployable" by Vocational Rehabilitation
    have successfully found employment or have started their own
    businesses after experiencing the ODEP programs.

-   **Micro-Enterprises** -- A number of customers of ODEP programs have
    turned to micro-enterprise as a form of employment by starting their
    own small business (with themselves as the only employee).

-   **Definition of Homelessness** - Although the Chronic Homelessness
    projects are still in their infancy, they report that the differing
    definitions of disability among government agencies have made it
    difficult for customers to obtain services.

-   **Sustainability of Services** -- A requirement for all of ODEP's
    demonstration projects is to ensure that the benefits gained will
    not disappear when ODEP funding ends. Sustainability will be
    achieved when customized employment becomes part of the menu of
    services available at the One Stop Career Centers (for the adult
    programs) and when providers recognize the complex needs of youth
    with disabilities and make changes to planning, policies and
    programs to address these needs (for the youth programs). To date,
    the project sites have reported that it could be difficult to
    sustain their projects without additional funding. Nevertheless,
    some sustainability has already been seen: Capacity-building
    activities at all sites (including training, improvements in
    accessibility, and increased collaboration among agencies and
    organizations) serve as a means of sustaining specific elements of
    the projects. Staff has been trained and will continue to receive
    training (primarily due to new competency requirements and formal
    training programs instituted at some sites). Collaborative
    partnerships will likely continue. Disability expertise has been
    built into many one Stop Career Centers. Finally, access to core
    services and One Stop Career Center programs has improved.

    **Technical Assistance**

The primary goal of the Technical Assistance program is supporting the
demonstration projects (adult, youth, and selected TA sites) in
achieving their goals and objectives. Each TA project has developed its
own approach to the provision of technical assistance based on the
nature of the programs it is supporting, the characteristics and needs
of the demonstration projects being supported, and the specific
strengths and experience of the partners implementing each TA project.

-   **National Center on Workforce and Disability for Adults" (NCWD/A)**
    and the **National Collaborative on Workforce and Disability for
    Youth" (NCWD/Y).** Since their inception in Fiscal Year 2001, the
    NCWD/A and NCWD/Y have seen changes in focus, structure,
    partnerships, and methods of operations. Now that both are
    approaching their last year of funding, they have evolved into
    useful vehicles for demonstration project sites, as well as the
    wider workforce development system. The NCWD/A has developed a
    systematic approach to providing TA to demonstration projects,
    including the development and implementation of a TA plan for each
    demonstration site. The NCWD/Y has produced a number of useful
    mechanisms for providing TA (e.g., the web site, provision of
    one-on-one TA, annual conferences, site visits, and sharing
    information with other demonstration projects).

-   **Training and Technical Assistance for Providers (T-TAP**). With a
    target audience of approximately 3,500 organizations throughout the
    United States, the T-TAP has been creative in developing methods for
    reaching its audience. In addition to the provision of regional
    seminars, the primary mechanism for training in T-TAP is electronic
    (e.g., with online seminars, webcasts, and online courses). Another
    creative way of accommodating its large and disparate audience is to
    collaborate with ODEP to identify community rehabilitation programs
    (CRPs) to receive the technical assistance. These organizations
    serve as demonstration projects for other programs that want to make
    the transition to promote integrated employment in non-stereotypical
    jobs for people with disabilities. The T-TAP also has developed a
    CRP. Leadership Network consisting of individuals affiliated with
    exemplary programs that have already converted or made significant
    progress in conversion to community-based employment and can serve
    as mentors for TA sites.

>  **The Chronic Homelessness Employment Technical Assistance Center
> (CHETA)**. Despite the disadvantage of beginning operations 13 months
> after the Chronic Homelessness projects were funded, CHETA has managed
> to use its collective partner expertise in homelessness, supported
> housing, and employment to provide useful technical assistance to
> Chronic Homelessness demonstration projects. During its first year of
> operation it has visited all of the Chronic Homelessness sites to
> conduct a needs assessment and conducted a week-long training session
> based on the findings from this needs assessment. The training session
> was geared toward Chronic Homelessness project staff and included
> panel discussions, workshops, role plays, and discovery sessions
> focused on housing and employment issues. The CHETA project consists
> of a partnership between two companies, one with expertise in
> supportive housing, and the other with expertise in employment issues.

**Conclusions**

**Summary**

Westat's independent evaluation continues to meet ODEP's three key
evaluation objectives of gauging program effectiveness, measuring
effectiveness to build workforce development system capacity, and
documenting systems change. During their external evaluation, Westat
developed key indicators for each of the systems change areas. Westat
then examined the inputs and processes developed by all demonstration
programs, and found that demonstration programs have segued from
building capacity to implementing a wide range of activities aimed at
making systems change.

The evaluation has also found that the programs heading towards the end
of their lifecycle are directing their efforts toward improving
processes to achieve improved outputs (e.g., more staff trained to
implement high quality customized employment services), as well as
engaging in activities to sustain their efforts. With assistance from
Technical Assistance Collaboratives, the older generation of programs
(i.e. programs funded in Fiscal Year 2001 and Fiscal Year 2002) is
paving the way for the younger generation (Fiscal Year 2003 and Fiscal
Year 2004) to move more quickly toward improving capacity of a workforce
development system to meet the employment and service needs of people
with disabilities.

**Next Steps**

The next stage of the evaluation will examine the self-evaluation
efforts of project sites to identify effective local, regional, and/or
state systems change efforts to build a sustained capacity to better
serve youths and adults with disabilities in achieving improved
employment and education outcomes.

In the past year, Westat has continued to conduct initial and followup
site visits, collected employment placement and retention data from
adult demonstration projects, and concentrated on a number of issues
that have arisen during the evaluation process (e.g., employer
involvement in demonstration programs; the role of SSA benefits in
program customers' employment decisions; and the role of intermediaries
in youth demonstration programs). In the next year of the evaluation,
Westat will focus on conducting final site visits to a number of
demonstration projects and follow-up visits to several others. These
visits will allow Westat the ability to better identify additional
promising practices and systems changes to the workforce development
system.

In addition to site visits, Westat will also obtain quantitative data
from project sites on competitive employment placements and retention in
order to provide ODEP with information on program outcomes.

[^1]: This *Research Brief* was developed from the interim report,
    [Evaluation of Disability Employment Policy Demonstration Programs:
    Task 10: Interim Report on ODEP Demonstration Programs:
    Accomplishments and Issues Identified by the Independent
    Evaluation](../DemoPrograms.pdf) (Westat, June 2005). This project
    is supported by Contract No. GS23F8144H, with funding from the U.S.
    Department of Labor, Office of Disability Employment Policy. The
    findings and recommendations of the report are those of the authors
    and should not in any way be considered or construed as official
    policy of the U.S. Department of Labor.