System Change as a Component of the External Evaluation 

of the Office of Disability Employment Policy (ODEP) 

Demonstration Program 
 

 

 

 

 

System Change as a Component of the External Evaluation of the Office of Disability Employment 

Policy Demonstration Program 

Focus Area Definitions 

As part of the external evaluation, Westat will examine changes to the workforce 

development system in six focus areas: (1) Capacity, (2) Coordination, (3) Customization (consumer 

choice and employer support), (4) Development/Adaptation/Evaluation of new practices, (5) 

Dissemination of effective practice, and (6) Sustainability of effective program elements. 1 

1. Capacity 

The Workforce Investment Act (WIA) of 1998 established a national workforce 

preparation and employment system to meet the needs of businesses, job seekers and those who want to 

further their careers through One-Stop single-entry access to job training, education, and employment 

services. While universal access is a guiding principle of the WIA, the One-Stop system and its staff 

often lack the capacity to provide coordinated, seamless service delivery to people with disabilities for the 

many programs and services which typically impact their entry or retention in the workforce. An overall 

objective of the ODEP Demonstration Program is to improve capacity throughout the entire workforce 

development system in order to increase employment and retention of people with disabilities. 

The capacity to achieve positive workforce outcomes includes changing perceptions, 

attitudes, and understanding of the issues related to disability, as well as improving the access and 

availability of people with disabilities to several different types of resources (e.g., funds, staff, time, 

resources, policies, procedures) and services. When examining system change within the category of 

capacity, the external evaluation will seek to determine whether there have been changes in: 

• The number and type of staff in the program 

• Staff background and training received 

•	 The ability of staff to work with disabled clients (i.e., can staff communicate with 

clients and do they have experience working with them?) 

1

 Note: Service delivery is another important component of the external evaluation and will be covered in other 

aspects of the evaluation (e.g., through extraction of data from Quarterly Reports). 
 

•	 The number, nature, target (e.g., staff, people with disabilities, family members, 

employers), and type of training programs offered 

•	 The number of training classes and number of people attending classes 

•	 Perceptions of stakeholders in the workforce development community (e.g., 

employers, business leaders, teachers, state officials) regarding the hiring of 

people with disabilities 

•	 Methods used to educate relevant stakeholders and One-Stop system personnel 

about delivering employment outcomes for people with disabilities 

•	 Sources of referrals for program services 

•	 Perceived barriers to employment in the community (e.g., outdated stereotypes 

and attitudes, lack of information) by people with disabilities and their families 

2. Coordination 

One of the barriers to positive employment outcomes among people with disabilities has 

been the lack of coordinated employment services for disabled people (i.e., services that are wellintegrated). Instead, services have existed in separate “silos” where the disabled person has traditionally 

had to move from one silo to another to obtain information and apply for and obtain services. The 

evaluation will seek to determine whether demonstration programs have changed the system in such a 

way that it is able to provide coordinated programs, processes, and services. Examples of variables that 

will be assessed for changes in coordination include: 

•	 Nature and extent of non-mandated partnerships 

•	 Number and nature of local partnerships formed (employers, workforce agencies) 

•	 Extent to which cross-agency training occurs 

•	 Extent to which inter-agency teams exist 

•	 Extent of interactions with other entities (e.g., employer groups, different levels 

of government, other non-profit organizations) 

•	 Roles, commitments, contributions (e.g., experience, funds, etc.) of each partner 

•	 Barriers to being able to fulfill commitments 

•	 Nature and extent of employer cooperation (e.g., type of occupations and 

industries available through grantee programs) 

•	 Nature of relationship and interaction among partners (formal/informal; in 

person/telephone; regular meetings/conferences) 

•	 Legislation/regulations/policies/practices of each partnering agency that impact 

on employment of people with disabilities 
•	 Sharing, leveraging, and exploitation of expertise and funds among partners and 

team members with local One-Stop Career Centers 

•	 Policies and procedures for coordinating funding, staffing, and grant activities 

with One-Stop Career Centers (e.g., flexible funding within and across the 

system) and access to other partners’ services (e.g., accommodations, 

technology) 

•	 Perceived barriers to employment in the community (e.g., transportation; lack of 

information) by people with disabilities and their families 

•	 Impact of performance measurement systems on outcomes for people with 

disabilities 

3. Customization (Consumer Choice and Employer Support) 

Instead of a one-size-fits-all approach, demonstration programs are being encouraged to 

customize (i.e., individualize) their programs and services in ways that will assist people with disabilities 

in seeking, obtaining, and retaining employment. Customization also applies to an employer’s approach 

to the placement of workers with disabilities, including workplace accommodations and job restructuring. 

The extent to which such customization has occurred is another focus of system change in the external 

evaluation. Factors that will explore customization include: 

•	 Involvement of the customer (e.g., people with disabilities, family members and 

potential employers), advocacy organizations, and other agencies in the planning 

stages of new or revised programs and/or service 

•	 The number and nature of new or revised programs and/or services and the extent 

to which such programs are customized 

•	 Use of such services and resources as SSI, SSDI, Ticket-to-Work, Medicaid, 

vocational rehabilitation, substance abuse counseling 

•	 Number and nature of different career choices available to people with 

disabilities through work-based experiences and career exploratory activities 

•	 The type of practices and services that have been developed/adapted (e.g., 

person-centered planning, personal budgeting, employment networks, eligible 

training providers, individual training accounts, vouchers, employment follow-up 

services) 

•	 Policies and procedures available to ensure customer choice and selfdetermination 
 

•	 Policies and procedures available to assist employers with customized 

employment options for people with disabilities. 

•	 Outreach to and communication with local employers to learn about their 

concerns and adapt services and programs to their needs 

•	 Ways in which programs address physical and programmatic accessibility (e.g., 

location, transportation, intake forms) 

•	 Procedures for tracking satisfaction of customers, employers, state and local 

officials, families, providers 

•	 Reported satisfaction with programs and services among people with disabilities, 

family members, employers, and others 

4. Development/Adaptation/Evaluation of New Practices 

ODEP wishes to ensure that its funding is being used to develop or adapt and evaluate 

new practices that are transportable to a variety of agencies and programs (as appropriate). The two 

important aspects to this focus area are: (a) developing or adapting innovative practices; and (b) 

determining whether they are effective. The external evaluation will be looking for changes in the system 

that are related to: 

•	 The type of practices and services that have been developed/adapted 

•	 Ways in which programs are delivered 

•	 The nature of self-evaluation techniques conducted by the demonstration 

program to identify effective practices 

•	 Ways in which information from self-evaluation has been used by the 

demonstration programs 

•	 Procedures for tracking satisfaction of customers, employers, state and local 

officials, families, providers 

•	 Reported satisfaction with programs and services 

5. Dissemination of effective practices 

The dissemination of effective practices addresses the propagation of new programs, 

practices, and processes throughout the workforce development system that work. The external 

evaluation will determine whether system elements have been changed for disseminating effective 

practices and whether dissemination has occurred. Indicators for examining dissemination include: 
•	 Promotion activities conducted by demonstration programs 

•	 Contents of websites 

•	 Use of the media and public forums 

•	 Networking and communications with other organizations 

•	 Use of alternative and accessible formats 

•	 Methods used to educate relevant stakeholders and One-Stop system personnel 

about delivering employment outcomes for people with disabilities 

6. Sustainability 

Once the capacity and coordination of the system have improved, new practices are being 

developed/adapted and evaluated, and effective practices are being disseminated, the logical outcome is 

expected to be a new, effective, and sustainable workforce development system. Sustainability is a 

gradual process that is not expected to occur immediately. Therefore, ODEP is interested in examining 

changes in the workforce development system that signal that sustainability is beginning to occur. 

Examples are changes in the: 

•	 Nature, extent, and sources of funding 

•	 Number and purpose of funding sources 

•	 Sharing, leveraging, and exploitation of expertise and funds from a variety of 

sources 

•	 Local and statewide legislation, regulations, policies and procedures that ensure 

that demonstration programs become part of the menu of services within the 

One-Stop 

•	 Changes in the local and state levels that help improve the situation for people 

with disabilities